Positional Encoding

Self-attention calculates relationships between tokens without any inherent notion of their order. Without positional encoding, the sentence 'The dog bit the man' would be processed identically to 'The man bit the dog.' Positional encoding solves this by adding a unique signal to each token's embedding based on its position, making the input representation order-aware before it enters the attention layers.

The original Transformer paper introduced fixed sinusoidal functions to encode absolute position. Each dimension of the positional vector corresponds to a sinusoid with a different frequency:

• Formula: PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
• Formula: PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
• Key Property: The linear relationship between positions allows the model to easily learn to attend by relative position, as PE(pos+k) can be represented as a linear function of PE(pos).

An alternative to fixed sinusoids is to treat position as a learnable parameter. A standard embedding lookup table is created for positions 0 up to the maximum context length. These embeddings are trained via backpropagation alongside all other model weights.

• Advantage: The model can learn an optimal encoding for the specific data distribution.
• Disadvantage: Cannot extrapolate to sequence lengths longer than those seen during training, unlike sinusoidal encodings which generalize to arbitrary lengths.

Instead of encoding absolute position, relative methods modify the attention mechanism itself to consider the pairwise distance between tokens. This directly models the intuition that the relationship between two words depends more on their distance than their absolute location.

• Approach: A learned bias is added to the attention score based on the offset between token i and token j.
• Benefit: Naturally generalizes to sequences longer than any seen during training, as only the relative distance matters.

RoPE encodes position by rotating the Query and Key vectors in multi-head attention by an angle proportional to their position. This elegantly combines absolute and relative position encoding.

• Mechanism: The dot product between Query and Key naturally decays with relative distance, encoding relative position implicitly.
• Adoption: Used in Meta Llama, Mistral, and many state-of-the-art open-source models due to its stability and ability to extend context length via interpolation techniques like NTK-aware scaling.

ALiBi removes positional embeddings entirely and instead adds a static, non-learned bias to the attention scores. This bias is a constant penalty proportional to the distance between tokens.

• Simplicity: No trainable positional parameters are required.
• Extrapolation: Demonstrates strong length extrapolation, performing well on sequences up to 2,000 tokens when trained on only 1,024.
• Implementation: For each attention head, a unique slope m is pre-defined, and the bias is computed as -m * |i - j|.

Two fundamental paradigms for injecting sequence order:

• Absolute Encoding: Assigns a unique vector to each position (e.g., sinusoidal or learned). Added directly to input embeddings before attention.
• Relative Encoding: Models the distance between token pairs during attention calculation. Transformer-XL and T5 use relative biases, while RoPE achieves this through vector rotation. Relative methods typically generalize better to sequences longer than those seen during training.

The original method from Vaswani et al. (2017) that uses sine and cosine functions of varying frequencies to generate deterministic position vectors. Each dimension of the encoding corresponds to a sinusoid with a different wavelength, forming a continuous geometric pattern. The key property is that the encoding for position pos + k can be represented as a linear function of the encoding for position pos, theoretically allowing the model to learn relative positions.

A simpler alternative where position representations are treated as trainable parameters in an embedding lookup table, identical to token embeddings. Used in early models like BERT and GPT-2, this approach lets the model discover optimal position representations during training. The primary limitation is the fixed maximum sequence length—positions beyond the pre-defined table size cannot be represented, requiring truncation or interpolation.

The fundamental limitation that makes positional encoding necessary. The self-attention mechanism computes weighted sums of Value vectors based solely on content similarity, with no inherent notion of token order. Without position information, the sentences 'The dog bit the man' and 'The man bit the dog' produce identical representations. Positional encoding breaks this symmetry by injecting sequence-order information into the otherwise order-agnostic attention calculation.

A relative encoding method that adds a static, non-learned bias to the attention scores before softmax. The bias is a linear penalty proportional to the distance between tokens, with different slopes per attention head. This simple approach eliminates the need for explicit position embeddings entirely and has demonstrated remarkable length extrapolation—models trained on short sequences can perform well on much longer ones without fine-tuning.