Causal Attention

The core mechanism is an upper triangular mask applied to the attention score matrix before softmax. This mask sets all scores where j > i to -∞, forcing the softmax output to zero for those positions.

• Implementation: A binary matrix M where M[i][j] = 0 if j <= i, else -∞
• Result: Token at position i can only attend to positions 0 through i
• Shape: An N x N matrix for a sequence of length N

The mask is applied element-wise: Attention(Q, K, V) = softmax((QK^T / √d_k) + M) * V

Causal attention enables token-by-token generation where each new token is conditioned solely on previously generated tokens. This is the foundation of modern language model inference.

• The model predicts P(token_i | token_0, ..., token_{i-1})
• Each generation step appends the new token to the sequence
• The process continues until an end-of-sequence token is produced
• Beam search and nucleus sampling are decoding strategies built on this property

Without causal masking, the model would cheat by peeking at future tokens during training.

Causal attention operates differently during training and inference, a critical distinction for ML engineers.

Training (Teacher Forcing):

• The full target sequence is fed as input
• The mask prevents attending to future positions
• All token predictions are computed in a single forward pass
• Loss is calculated across all positions simultaneously

Inference (Autoregressive Decoding):

• Tokens are generated one at a time
• The KV Cache stores previous Key and Value vectors
• Each step only computes attention for the newest token
• Memory grows linearly with sequence length

Causal attention is a specific form of masked self-attention, but not all masked attention is causal.

• Causal Masking: Strictly upper triangular; enforces temporal ordering
• Padding Masking: Masks out <PAD> tokens to prevent attending to meaningless positions
• Sparse Masking: Uses predefined patterns like sliding windows or block-sparse structures
• Custom Masking: Can enforce arbitrary constraints like document boundaries

In practice, causal and padding masks are often combined into a single additive mask before the softmax operation.

The Key-Value Cache is the primary inference optimization enabled by causal attention. Since past tokens never change, their Keys and Values can be stored and reused.

• Without cache: Recompute attention for all tokens at every step — O(n²) per step
• With cache: Only compute K, V for the new token — O(n) per step
• Memory cost: 2 * num_layers * num_heads * head_dim * sequence_length floats
• Variants: Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce cache size by sharing K and V heads

This optimization is what makes real-time text generation feasible.

Causal attention is the defining characteristic of decoder-only Transformer architectures like GPT, Llama, and PaLM. Unlike encoder-decoder models, these architectures rely entirely on causal self-attention.

• No separate encoder: The model processes input and generates output in one stack
• No cross-attention: All attention is self-attention within the same sequence
• Pre-training objective: Next-token prediction (causal language modeling)
• Advantage: Simpler architecture, unified pre-training, strong zero-shot performance

This design choice has proven remarkably effective, driving the modern LLM revolution.

A Transformer configuration consisting solely of decoder blocks with causal self-attention, used by modern autoregressive language models like GPT to generate text token by token. Unlike encoder-decoder models, there is no separate encoding stage—the same stack processes the entire sequence left-to-right. Causal masking is the defining feature that prevents each token from attending to future positions, enforcing the autoregressive property required for next-token prediction during both training and generation.

An inference optimization technique that stores the previously computed Key and Value vectors for all generated tokens, preventing redundant re-computation during autoregressive decoding. Without a KV cache, each new token would require re-computing attention over the entire sequence, leading to quadratic growth in computation. With caching, only the new token's Query, Key, and Value are computed, and the stored Keys and Values from prior steps are reused. This is essential for causal attention during generation.

A variant of multi-head attention where all attention heads share a single set of Key and Value projection weights, dramatically reducing the size of the KV cache and memory bandwidth requirements during inference. In standard multi-head attention, each head has its own K and V projections, multiplying cache size by the number of heads. MQA trades some model quality for significant inference speedups, making it particularly valuable for causal attention in latency-sensitive deployments.

An interpolation between multi-head and multi-query attention that partitions Query heads into groups, with each group sharing a single set of Key and Value heads. This balances inference speed and model quality—fewer KV heads than Query heads reduces cache size, while maintaining more expressiveness than pure MQA. GQA has been adopted in models like Llama 2 and Llama 3 as the default attention mechanism, directly impacting how causal attention scales during autoregressive generation.

An N × N matrix where each entry represents the raw compatibility score between a Query and a Key token, computed before softmax normalization. In causal attention, this matrix is modified by applying an upper-triangular mask that sets all positions where the column index exceeds the row index to negative infinity. After softmax, these positions become zero, ensuring token i can only attend to tokens 1 through i. This matrix is the mathematical instantiation of the autoregressive constraint.

A sparse attention pattern where each token attends only to a fixed-size window of neighboring tokens, reducing complexity from O(n²) to O(n) with respect to sequence length. When combined with causal masking, each token attends to the W tokens immediately preceding it. This approach, used in models like Mistral, enables processing of very long documents while maintaining the autoregressive property. Layers can be stacked with different window sizes or combined with global attention for select tokens.