Positional Encoding

Assigns a unique vector to each position in the sequence using deterministic mathematical functions. The original Transformer uses sinusoidal functions of varying frequencies:

• Formula: PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
• Formula: PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

This allows the model to extrapolate to sequence lengths not seen during training, as the functions are continuous and bounded.

Treats each position index as a discrete token and learns a dedicated embedding vector during training, identical to how word embeddings are learned.

• BERT and GPT models use this approach
• Vectors are stored in a lookup table of size [max_seq_len, d_model]
• Cannot extrapolate beyond the maximum training length
• Often outperforms sinusoidal encoding on fixed-length tasks due to task-specific optimization

Encodes the pairwise distance between tokens rather than their absolute positions. This captures the intuition that the relationship between words depends more on their relative offset than their absolute location.

• Transformer-XL uses sinusoidal relative encodings
• T5 employs learned relative position biases per attention head
• Enables better generalization to sequences longer than those seen during training
• Critical for tasks where local context patterns dominate

Encodes position by rotating the query and key vectors in the attention mechanism by an angle proportional to their position index. The dot product between rotated vectors naturally depends only on their relative distance.

• Used in LLaMA, Mistral, and PaLM architectures
• Provides theoretical elegance: relative position emerges from absolute rotations
• Decays attention scores with increasing relative distance
• Supports efficient long-context extrapolation with minimal modification

Removes positional embeddings entirely and instead subtracts a static, non-learned bias from the attention scores proportional to the distance between tokens. Each attention head uses a different slope.

• Simpler than RoPE or sinusoidal methods
• Used in BLOOM and early efficient Transformer variants
• Demonstrates strong length extrapolation: a model trained on 1024 tokens can perform well on 2048+
• The bias is computed as: bias = -m * |i - j| where m is a head-specific slope

Extends positional encoding beyond 1D sequences to handle multi-dimensional data like images, video, and point clouds. Position is encoded along multiple axes simultaneously.

• Vision Transformers (ViT) use learned 2D patch position embeddings
• Video models encode (time, height, width) as separate or combined embeddings
• Spatial coordinates in point cloud processing use sinusoidal encodings across X, Y, Z dimensions
• Enables Transformers to process non-sequential structured data

A neural network design that processes entire sequences in parallel rather than sequentially. Unlike recurrent models, Transformers have no inherent sense of token order, making positional encoding a critical component. The architecture stacks multiple self-attention and feed-forward layers, enabling it to capture long-range dependencies without the vanishing gradient problems that plague RNNs. Introduced in the 2017 paper Attention Is All You Need, it now underpins virtually all modern large language models.

A mechanism that computes a contextual representation for each token by weighing its relevance against every other token in the sequence. For each position, the model calculates query, key, and value vectors, then produces an output as a weighted sum of values. Critically, self-attention is permutation-invariant—without positional encoding, shuffling the input sequence produces identical outputs, destroying syntactic and semantic order.

The original method proposed in the Transformer paper, using fixed sine and cosine functions of varying frequencies to encode position. Each dimension of the encoding vector corresponds to a sinusoid with a different wavelength:

• Formula: PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
• Formula: PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)) This design allows the model to easily learn relative positions, as the linear relationship between any two positions can be expressed through the sinusoid properties.

An alternative approach where position representations are treated as trainable parameters initialized randomly and updated via backpropagation. Each position index gets its own embedding vector, identical in dimension to token embeddings. Common in models like BERT and GPT, learned embeddings offer flexibility but are limited to a maximum sequence length defined at training time. They cannot naturally extrapolate to positions beyond this fixed window without interpolation or special techniques like ALiBi.

A widely adopted method that encodes position by rotating the query and key vectors in self-attention by an angle proportional to their position. Key properties:

• Relative position is captured through the dot product of rotated vectors
• Absolute position decays naturally with token distance
• Enables length extrapolation beyond training context windows Used in LLaMA, Mistral, and Gemma model families, RoPE has become the dominant encoding scheme in modern open-source LLMs due to its mathematical elegance and empirical performance.

A family of methods that model pairwise distances between tokens rather than absolute positions. Instead of assigning a position ID to each token, these encodings inject a bias into the attention computation based on the offset between token pairs:

• T5 uses learned scalar biases for each relative distance bucket
• Transformer-XL uses sinusoidal relative encodings for recurrence
• DeBERTa disentangles content and position in attention Relative encodings naturally handle sequences longer than those seen during training.