Causal Attention Mask

The mask is a square matrix where the upper triangle is set to -infinity (or a very large negative number) and the lower triangle is set to 0. This forces the softmax function to assign zero probability to future positions.

• Shape: [seq_len, seq_len] for each attention head
• Upper Triangle: Masked (prevented from attending)
• Lower Triangle & Diagonal: Unmasked (allowed to attend)
• Implementation: Added to attention scores before softmax: scores = scores + mask

The mask ensures that the prediction for token at position i depends only on tokens at positions 0 through i-1. This is the mathematical implementation of the chain rule of probability in sequence modeling.

• Token t can only attend to tokens [0, 1, ..., t]
• Token t+1 is completely invisible during the prediction of token t
• Enables teacher forcing during training: the model learns to predict the next token given all previous ground-truth tokens

The mask operates differently depending on the phase:

Training (Teacher Forcing)

• The full sequence is fed in parallel
• The mask prevents information leakage from future tokens
• All positions are computed simultaneously via matrix multiplication

Inference (Autoregressive Decoding)

• Tokens are generated one at a time
• The KV-Cache stores previous Key and Value tensors
• The mask grows incrementally with each new token generated

The causal mask is applied to the scaled dot-product attention scores before the softmax normalization. Without it, the model would have bidirectional context and could cheat by looking at the answer.

• Attention formula: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k) + M) * V
• M is the causal mask matrix
• After softmax, masked positions have weight 0.0
• The mask is applied independently in each attention head across all layers

The causal mask is often confused with the padding mask, but they serve different purposes:

• Causal Mask: Prevents attending to future tokens (temporal constraint)
• Padding Mask: Prevents attending to <PAD> tokens in batched sequences of different lengths
• Both are combined in practice: combined_mask = causal_mask & padding_mask
• Padding masks are typically 1 for real tokens and 0 for padding tokens

The causal attention mask is the defining characteristic of decoder-only transformers like GPT, LLaMA, and Claude. It distinguishes them from encoder-only models like BERT which use bidirectional attention.

• GPT Family: Causal self-attention in every layer
• Encoder-Decoder Models (T5, BART): Causal mask only in the decoder cross-attention
• Prefix Language Models: Hybrid approach with bidirectional attention on the prefix and causal attention on the generated portion

A binary tensor applied during the self-attention mechanism to control which tokens can interact. It sets attention weights to negative infinity before the softmax, effectively zeroing them out.

• Padding Mask: Prevents the model from attending to meaningless padding tokens in batched inputs.
• Causal Mask: A specific triangular matrix that prevents attending to future tokens.
• Applied at every transformer layer during both training and inference.

A sequence modeling paradigm where the prediction of the next token is conditioned solely on previously generated tokens. The causal attention mask is the architectural enforcement of this principle.

• Each token is generated one at a time, left to right.
• The output at step t becomes part of the input at step t+1.
• Contrasts with bidirectional attention used in encoder models like BERT.

A memory optimization that stores the Key and Value tensors of previously computed tokens during autoregressive decoding. Without a causal mask, the KV-cache would be invalidated by bidirectional contamination.

• Eliminates redundant computation for past tokens.
• Transforms inference from quadratic to linear complexity per step.
• The causal mask ensures the cache remains a faithful representation of the prefix.

The dominant training strategy for autoregressive models where the ground-truth previous token is fed as input, regardless of the model's own prediction. The causal mask makes this parallelizable.

• The entire target sequence is shifted right by one position.
• The mask allows the model to compute loss for all positions simultaneously.
• Prevents compounding of early errors during the training phase.

The core operation where every token in a sequence computes a weighted representation of all other tokens. The causal mask modifies this to be unidirectional.

• Computes Query, Key, and Value vectors for each token.
• The attention score matrix is element-wise added to the mask.
• Without the mask, this is bidirectional self-attention; with it, it becomes masked self-attention.

The maximum span of tokens a model can attend to. The causal mask operates strictly within this boundary, defining the limit of the model's immediate working memory.

• Modern models range from 4K to 1M+ tokens.
• The mask is a square matrix of size [window_size x window_size].
• Truncation strategies must respect the causal structure to avoid logical breaks.