Attention Mask

A causal mask (or autoregressive mask) is a triangular matrix that enforces a unidirectional flow of information. It ensures each token can only attend to previous tokens and itself, preventing the model from 'seeing the future.' This is the standard mask used in decoder-only models like GPT for autoregressive text generation. The mask is typically applied by setting future positions to -inf before the softmax operation, forcing their attention weights to zero.

• Key Property: Preserves the sequential, causal nature of language.
• Implementation: Upper-triangular matrix of -inf (or very large negative) values.
• Example: In the sequence [A, B, C], token C can attend to A, B, and C, but token A cannot attend to B or C.

A padding mask is used during batch processing to ignore padding tokens added to standardize sequence lengths. It prevents the model from attending to these meaningless tokens, which would dilute the attention signal and waste computation. The mask is applied by setting the attention scores for padding token positions to -inf before the softmax.

• Key Property: Enables efficient batch training and inference.
• Implementation: Binary mask where 1 indicates a real token and 0 indicates a pad token.
• Example: For a batch with sequences [[cat, dog, <PAD>], [bird, <PAD>, <PAD>]], the mask ensures attention is only calculated for cat, dog, bird.

In encoder models like BERT, the standard self-attention mask is a full attention mask. It allows every token in the input sequence to attend to every other token, enabling a bidirectional understanding of context. This is fundamental for tasks like text classification and named entity recognition, where the meaning of a word depends on its entire surrounding context.

• Key Property: Enables full, bidirectional context understanding.
• Implementation: A matrix of all ones (or zeros for -inf).
• Contrast: Differs fundamentally from the causal mask used in decoders.

A cross-attention mask controls the flow of information between two distinct sequences in encoder-decoder architectures (e.g., T5, BART). It governs which tokens from the encoder's output (the source) the decoder can attend to at each generation step. This is crucial for sequence-to-sequence tasks like translation and summarization.

• Key Property: Manages information flow from a source context to a generating target.
• Implementation: Often a full mask from decoder query to encoder keys/values, but can be constrained (e.g., for document grounding).
• Function: Allows the decoder to 'look back' at the encoded source sequence while generating the output.

Beyond standard types, custom attention masks are engineered to impose specific, often sparse, connectivity patterns. This includes:

• Sliding Window Masks: Restrict attention to a local neighborhood of tokens to enable processing of very long sequences.
• Block-Sparse Masks: Allow attention only between predefined blocks of tokens, reducing quadratic complexity.
• Task-Specific Masks: For example, masking out certain segments of a document to simulate a fill-in-the-blank task or enforcing graph-based relationships between tokens.

These are central to efficient transformer variants like Longformer and BigBird, which trade off some global context for the ability to handle much longer inputs.

In real training and inference, masks are often combined using element-wise addition. The most common combination merges a causal mask with a padding mask. The logic is additive: a position is masked if either the causal rule or the padding rule dictates it should be ignored. The combined mask tensor is added to the attention scores before the softmax, where masked positions are set to a large negative value to zero out their probability.

• Workflow: 1) Calculate raw attention scores (Q•K^T). 2) Add the combined mask tensor. 3) Apply softmax.
• Result: The model simultaneously respects the autoregressive generation order and ignores computational padding.
• Example: In a batched generation task, the final mask for each sequence is the logical OR of its causal and padding masks.

The causal mask (or autoregressive mask) is a triangular matrix that enforces the unidirectional constraint essential for text generation. It ensures each token can only attend to previous tokens and itself, preventing the model from 'seeing the future' during autoregressive decoding. This is implemented by setting future positions to -inf (effectively zero after softmax).

• Implementation: A lower-triangular matrix of ones.
• Use Case: All autoregressive decoder-only models like GPT-3, LLaMA, and GPT-4 use this for next-token prediction.

A padding mask handles variable-length sequences within a single batch. Shorter sequences are padded with a special token (e.g., [PAD]), and the mask prevents the model from attending to these meaningless padding tokens, which would dilute the attention scores.

• Mechanism: The mask sets attention weights to -inf for all [PAD] token positions.
• Impact: Enables efficient GPU utilization by processing multiple sequences of different lengths simultaneously without computational waste or information leakage.

In models like T5 or BART, distinct masks govern the encoder and decoder blocks. The encoder uses a full attention mask, allowing each token to attend to all others in the input. The decoder uses a causal mask for its own output and a cross-attention mask to selectively attend to the encoder's final hidden states.

• Cross-Attention: The decoder's queries attend to the encoder's keys and values, with the mask controlling which encoder tokens are accessible for each decoding step.

Masks can enforce task-specific attention patterns. For example, in JSON or SQL generation, a mask can restrict a model to only attend to relevant schema keys or column names when generating a value. In few-shot prompting, a mask can isolate the demonstration examples from the query to prevent leakage.

• Example: A system prompt defining a role can be masked to prevent the model from attending to user instructions meant for a different part of the workflow.

For processing sequences longer than the standard context window, sparse attention patterns like sliding window attention are implemented via specialized masks. These masks limit each token's attention to a local neighborhood, reducing computational complexity from O(n²) to O(n*k).

• BigBird & Longformer: Use a combination of global, window, and random attention patterns defined by fixed masks to handle documents of up to 4096+ tokens.

During fine-tuning, masks are crucial for contrastive learning objectives like those used in RLHF. For a given prompt, the positive completion and negative (rejected) completion are processed in parallel. A mask ensures the model computes loss only on the token differences between the two sequences, isolating the impact of the qualitative divergence.

The context window is the fixed-size, contiguous block of tokens a transformer model can process in a single forward pass. It acts as the model's working memory for a given input and output sequence. The attention mask directly operates within this bounded space.

• Fixed Limit: Defines the maximum sequence length (e.g., 128K tokens).
• Working Memory: All contextual understanding for a generation is derived from tokens within this window.
• Attention Scope: The self-attention mechanism, guided by the mask, computes relationships only between tokens present in the current window.

Positional Encoding (or embeddings) injects information about the absolute or relative order of tokens into the model's input. Since the self-attention mechanism is permutation-invariant, this is essential for understanding sequence structure.

• Absolute vs. Relative: Methods like sinusoidal functions or learned embeddings provide positional cues.
• Interaction with Masking: The attention mask controls which tokens attend to each other, while positional encoding informs the model where those tokens are in the sequence.
• Long Context Challenges: Techniques like Rotary Positional Embedding (RoPE) help generalize to sequences longer than those seen in training.

The KV Cache is a critical inference optimization that stores computed key and value vectors for previous tokens during autoregressive generation. This prevents redundant computation, drastically improving speed for long sequences.

• Cache Management: The attention mask determines which cached key-value pairs are relevant for the current generation step.
• Memory Footprint: The cache grows linearly with sequence length, creating memory pressure for long contexts.
• Eviction Policies: Techniques like cache eviction (e.g., LRU) are needed to manage the cache when memory is full, directly interacting with attention masking strategies.

Sparse Attention is a class of transformer architectures that restrict the full self-attention mechanism to a predefined subset of token pairs. This reduces computational complexity from quadratic to near-linear for very long sequences.

• Architectural Masking: Implements a fixed, often learnable, attention mask pattern (e.g., local windows, strided patterns, global tokens).
• Efficiency vs. Expressivity: Trades off the model's ability to see all pairwise connections for the ability to process much longer contexts.
• Examples: Sliding Window Attention, where a token only attends to a fixed number of preceding tokens, is a common sparse pattern.

Context Compression refers to techniques that reduce the token footprint of information within the context window while attempting to preserve semantic utility. It's a higher-level strategy for managing limited context.

• Techniques: Includes context summarization, redundancy elimination, and selective pruning.
• Relationship to Masking: Compression often creates a new, shorter sequence. A standard attention mask is then applied to this compressed context.
• Goal: To free up token budget for new inputs or longer outputs without simply truncating earlier information.

A Causal Mask (or autoregressive mask) is a specific type of attention mask that enforces a unidirectional flow of information. It is fundamental to decoder-only models used for text generation.

• Mechanism: Allows a token to attend only to itself and preceding tokens in the sequence, preventing "peeking" at future tokens.
• Implementation: A triangular matrix of 1s (allow) and 0s (block), typically applied to the attention scores before the softmax.
• Contrast with Padding Mask: A causal mask is applied uniformly for modeling, while a padding mask handles variable-length sequences in a batch.