Positional Encoding
Attention is permutation-equivariant: permute the columns of its input and its output is permuted the same way. On its own it therefore cannot tell one ordering of a sequence from another. The original transformer paper [33] resolves this by adding a fixed matrix to the input, whose columns encode the position:
\[P_{2j+1,\,i} = \sin\left(\frac{i - 1}{10000^{2j/d}}\right), \qquad P_{2j+2,\,i} = \cos\left(\frac{i - 1}{10000^{2j/d}}\right),\]
with $d$ the feature dimension. Each pair of rows encodes the position at one wavelength, and the wavelengths grow geometrically from $2\pi$ towards the $10000\cdot2\pi$ of the paper, which they approach only as $d$ grows — at $d = 4$ the longest is $100\cdot2\pi$. Distant positions are therefore distinguished by the slow rows and neighbouring ones by the fast rows.
Why it is off by default
The transformers in this package are usually applied to phase-space trajectories, where the ordering is carried by the data rather than needing to be encoded: a TransformerIntegrator is handed a time series whose columns are consecutive time steps. So Transformer takes positional_encoding = false unless asked otherwise.
It does not disturb the geometry
The layer adds a constant, so its Jacobian is the identity. A structure-preserving architecture with this layer in front therefore preserves exactly what it preserved without it — a LinearSymplecticTransformer stays symplectic, a VolumePreservingTransformer stays volume-preserving. The test suite asserts the Jacobian claim directly; the two compositions follow from it.
The layer holds no parameters, and the sequence length is read from the input at every call rather than fixed when the layer is built, because a network here is applied to trajectories of whatever length the data has.
GeometricMachineLearning.PositionalEncoding — Type
PositionalEncoding(dim)A layer that adds the sinusoidal positional encoding of [33] to its input.
It has no parameters. The sequence length is read from the input's second axis at every call rather than fixed at construction, because a transformer here takes the sequence length from its data and not from its architecture — the same network is applied to trajectories of different lengths. The encoding matrix is therefore built per call, which costs one dim × seq_length allocation, and a second for the sum.
positional_encoding builds a Matrix, so this layer adds a host array to whatever it is given. Every other layer here allocates through the backend — KernelAbstractions.allocate, or similar(x, …) in the forward pass — and this one does not yet. On a GPU array this is not a slowdown but a failure: the broadcast fails to compile, because the host array cannot be read from a kernel. A network carrying this layer is therefore a CPU network. Nothing in the test suite would catch that, because the suite has no GPU test.
Its Jacobian is the identity, since it adds a constant, so it composes with the structure-preserving architectures without changing what they preserve: a LinearSymplecticTransformer with this layer in front is as symplectic as one without it.
See positional_encoding for the matrix itself, and Transformer, whose positional_encoding keyword puts this layer at the front of the chain.
Examples
using GeometricMachineLearning
l = PositionalEncoding(4)
x = zeros(Float32, 4, 3)
l(x, NamedTuple())[:, 1]
# output
4-element Vector{Float32}:
0.0
1.0
0.0
1.0GeometricMachineLearning.positional_encoding — Function
positional_encoding(T, dim, seq_length)The sinusoidal positional encoding of [33], as a dim × seq_length matrix:
\[P_{2j+1,\,i} = \sin\left(\frac{i - 1}{10000^{2j/d}}\right), \qquad P_{2j+2,\,i} = \cos\left(\frac{i - 1}{10000^{2j/d}}\right),\]
with $d$ the feature dimension dim. Row pairs share a frequency, and the frequencies decay geometrically, so a pair of rows encodes the position at one wavelength. The shortest wavelength is $2\pi$; the longest approaches the $10000\cdot2\pi$ of the paper only as $d$ grows, and at $d = 4$ it is $100\cdot2\pi$.
Positions are counted from zero — the paper writes only "pos is the position" and gives no base, and zero is the standard reading — so the first column is $(0, 1, 0, 1, \ldots)$.
Examples
using GeometricMachineLearning
positional_encoding(Float32, 4, 3)[:, 1]
# output
4-element Vector{Float32}:
0.0
1.0
0.0
1.0References
- [33]
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser and I. Polosukhin. Attention is all you need. Advances in neural information processing systems 30 (2017).