Attention Mask

Prevents the model from attending to padding tokens in batched sequences of unequal length. Without this mask, the model would waste computation on empty positions and allow padding embeddings to corrupt the attention softmax.

• Applied as an additive mask (large negative value) before softmax
• Ensures padding tokens contribute zero weight to the output
• Critical for efficient batched inference where sequences are padded to uniform length
• Example: In a batch with sequences of lengths 5, 12, and 8, all are padded to 12; the mask zeros out positions 6-12 for the first sequence

A lower triangular matrix that enforces autoregressive generation by preventing each token from attending to future positions. This is the defining characteristic of decoder-only architectures like GPT.

• Token at position i can only attend to positions 0 through i
• Implemented as an upper-triangular mask filled with negative infinity
• Enables parallelized training despite sequential generation at inference
• Without this mask, the model would cheat by peeking at the answer during next-token prediction

A dynamic mask that restricts attention to a sliding window or fixed span, used when processing documents that exceed the model's maximum context length.

• Implements local attention patterns to handle arbitrarily long sequences
• Often combined with global tokens that can attend to the full sequence
• Reduces computational complexity from O(n²) to O(n × w) where w is window size
• Example: Longformer and BigBird use this to process 4,096+ token documents efficiently

Controls how the decoder attends to encoder outputs in encoder-decoder architectures like T5 and BART. Each decoder position can attend to the full encoder output sequence.

• Queries come from the decoder, keys and values from the encoder
• No causal restriction on the encoder side — full bidirectional access
• Enables the model to condition generation on a complete understanding of the input
• Distinct from self-attention masks which operate within a single sequence

Predefined mask structures that limit each token to attending to a subset of positions rather than the full sequence, dramatically reducing memory footprint for long contexts.

• Strided pattern: Attend to every nth token plus local neighbors
• Block-sparse: Divide sequence into blocks; tokens attend within their block and to select global tokens
• Random pattern: Each token attends to a random subset, approximating full attention
• Enables processing of sequences up to 1 million tokens on a single GPU

Domain-specific masks that enforce structural constraints based on external knowledge, such as preventing cross-attention between unrelated entities in a knowledge graph.

• Used in graph-aware Transformers for molecular modeling and code understanding
• Masks can encode syntactic parse trees, ontology hierarchies, or molecule bonds
• Prevents the model from forming spurious correlations between unrelated concepts
• Example: In code generation, masking prevents a function from attending to variables outside its scope

A triangular matrix that enforces the autoregressive property by preventing the model from attending to future tokens. During training, the mask sets all attention scores above the diagonal to negative infinity before the softmax operation, ensuring the prediction for token t depends only on tokens 0 through t-1. This is the specific instantiation of an attention mask used in decoder-only models like GPT. Without it, the model would cheat by peeking at the answer.

A binary tensor that prevents the model from wasting computation on meaningless padding tokens. Since inputs within a batch must have uniform length, shorter sequences are padded with a special <PAD> token. The padding mask sets the attention score for these positions to a large negative number, ensuring the softmax output is effectively zero. This is distinct from the causal mask: it filters out static filler rather than enforcing temporal order.

A memory optimization that stores the Key and Value tensors of previously generated tokens to avoid recomputing them at each autoregressive step. During generation, the causal attention mask is dynamically constructed to allow the new query token to attend to all cached keys and values. The mask ensures the query cannot see future queries that haven't been generated yet, while the cache eliminates redundant computation for the static prefix.

The core operation where the attention mask is applied. The mechanism computes Query (Q), Key (K), and Value (V) matrices from the input. The raw attention scores are calculated as the dot product of Q and K, scaled by the square root of the dimension. The attention mask is added element-wise to these scores before the softmax. This additive approach ensures masked positions receive a probability of zero, effectively removing them from the weighted sum of values.

The maximum span of tokens a model can attend to, defining the absolute boundary of the attention mask. The mask's dimensions are directly constrained by this limit. When the input sequence exceeds the context window, truncation strategies must decide which tokens to drop. The attention mask then operates only on the surviving tokens, making the window size a hard architectural constraint that dictates the mask's maximum shape.

Used in encoder-decoder architectures like T5, this mask controls how the decoder attends to the encoder's output. Unlike self-attention masks, cross-attention masks do not need to be causal because the decoder can attend to the entire encoded input sequence simultaneously. However, padding masks are still applied to the encoder output to ignore filler tokens. This allows the decoder to condition generation on the full source context.