ALiBi (Attention with Linear Biases)

ALiBi subtracts a constant bias from the attention score for each pair of tokens. The penalty increases linearly with the distance between the query and key.

• Formula: softmax(q_i K^T + m * [-(i-1), ..., -2, -1, 0])
• The slope m is a head-specific scalar that is geometrically spaced across attention heads.
• Closer tokens receive a smaller penalty, naturally encouraging the model to prioritize local context.
• Unlike sinusoidal or learned embeddings, this bias is not added to the word embedding; it is applied directly inside the attention computation.

ALiBi requires absolutely no trainable parameters for position encoding. The slopes m are pre-computed constants.

• Slope Calculation: For n heads, slopes form a geometric sequence starting at 2^(-8/n) and ending at 2^(-8).
• This eliminates the need to train or store a position embedding matrix, reducing model size and complexity.
• The static nature ensures the model's behavior on long sequences is deterministic and not dependent on how far the training data extended.
• This contrasts sharply with Rotary Position Embedding (RoPE) and learned absolute position embeddings, which require careful interpolation for extrapolation.

ALiBi's primary advantage is the ability to perform inference on sequences far longer than those seen during training without fine-tuning.

• A model trained on 1024 tokens can reliably generalize to 2048, 4096, or even longer sequences.
• The linear bias provides a smooth, monotonic inductive bias that does not break down at unseen lengths.
• This directly addresses the Lost-in-the-Middle problem by providing a consistent structural signal regardless of absolute position.
• Empirical results show ALiBi maintains lower perplexity on extrapolated lengths compared to sinusoidal or RoPE embeddings without any interpolation tricks.

The bias computation is extremely lightweight and can be fused into the attention kernel with negligible overhead.

• The bias mask is a simple Toeplitz matrix that can be pre-computed and cached.
• No additional matrix multiplications or rotations are required, unlike RoPE which applies complex rotations to query and key vectors.
• This makes ALiBi particularly attractive for edge deployment and high-throughput inference scenarios.
• The memory footprint for the bias is O(sequence_length), which is trivial compared to the O(sequence_length * hidden_dim) KV-Cache.

For autoregressive decoder models, ALiBi naturally integrates with the causal attention mask to enforce temporal order.

• The linear penalty is applied only to the lower-triangular portion of the attention matrix.
• Future tokens remain masked with -inf while past tokens receive the distance-based penalty.
• This dual masking ensures the model cannot attend to future information while still prioritizing recent context over distant history.
• The combination creates a strong recency bias that mirrors how many natural language phenomena exhibit local coherence.

Each attention head receives a unique slope value, creating a diverse set of receptive fields across the model.

• Shallow slopes (small m) allow some heads to attend broadly across the entire context.
• Steep slopes (large m) force other heads to focus narrowly on immediate neighbors.
• This geometric distribution ensures the model can simultaneously capture both short-range syntax and long-range semantics.
• The diversity is fixed and non-learned, preventing any single head from dominating and ensuring robust generalization.

The dominant alternative to ALiBi. RoPE encodes position by rotating the query and key vectors in the attention computation, enabling the model to naturally capture relative positional relationships. Unlike ALiBi's static linear penalty, RoPE uses sinusoidal functions to modulate the dot-product attention score based on the relative distance between tokens. This method has become the standard for most state-of-the-art open-source models (Llama, Mistral) due to its strong context length extrapolation properties when combined with NTK-aware scaling.

The core problem ALiBi solves. This is the ability of a model to perform inference on sequences longer than those seen during training without catastrophic performance degradation. Standard learned position embeddings fail completely on unseen lengths. ALiBi achieves extrapolation by design: its linear bias penalty is a simple mathematical function of distance, not a learned parameter, so it generalizes to any sequence length without modification. This allows a model trained on 1024 tokens to reliably process 2048+ tokens at inference.

The architectural component ALiBi modifies. In a standard Transformer, the attention score between a query and key is computed as a scaled dot-product. ALiBi intervenes by subtracting a static, non-learned bias from this score before the softmax:

• The bias is proportional to the distance between tokens
• Each attention head gets a unique slope
• Closer tokens receive less penalty; distant tokens receive more This enforces a recency bias that mimics the natural locality found in language.

The memory buffer that stores pre-computed Key and Value tensors from previous decoding steps to avoid redundant computation during autoregressive generation. ALiBi's simplicity directly benefits KV-Cache management:

• No position-dependent computations in the cache
• The linear bias is applied on-the-fly during attention scoring
• This means cached KV pairs are position-agnostic, simplifying batching and memory allocation Efficient KV-Cache usage is critical for low-latency, high-throughput LLM serving.

A documented failure mode of LLMs where information placed in the center of a long context window is significantly less likely to be retrieved than information at the beginning (primacy bias) or end (recency bias). ALiBi's linear penalty structure inherently reinforces a strong recency bias, which can exacerbate this effect for middle-context tokens. Understanding this trade-off is essential: ALiBi excels at extrapolation but may require careful prompt engineering to ensure critical instructions are placed near the beginning or end of the context window.

An exact-attention algorithm that dramatically speeds up computation and reduces memory footprint by minimizing HBM reads/writes through tiling and recomputation. ALiBi is fully compatible with FlashAttention because the linear bias can be fused into the attention kernel as a simple element-wise operation. This combination—ALiBi for length extrapolation and FlashAttention for hardware efficiency—enables fast inference on extremely long sequences without the memory overhead of learned position embeddings or complex RoPE calculations.