Attention Mask

An attention mask is a tensor of boolean values where 1 indicates a genuine content token and 0 indicates a padding token. During the scaled dot-product attention calculation, positions corresponding to 0 are set to a large negative value (e.g., -1e9) before the softmax function. This forces the attention weights for those positions to effectively zero, ensuring the model's focus is strictly constrained to real linguistic content and not wasted on empty computational slots.

In decoder-only architectures like GPT, the attention mask takes the form of an upper triangular matrix filled with negative infinity. This prevents a token at position i from attending to any token at position j where j > i. By masking out future tokens, the model is forced to predict the next word based solely on the preceding context, preserving the autoregressive property essential for coherent text generation without information leakage from the future.

Since neural networks process data in fixed-size batches, sequences of varying lengths must be standardized. A padding mask is dynamically generated by the tokenizer during the collation step. The mask marks all positions beyond the original sequence length as 0. This allows the model to process a batch containing a 10-word sentence and a 50-word sentence simultaneously without the shorter sequence's padding tokens distorting the attention distribution or the final contextualized representations.

For tasks like sentiment analysis, the attention mask plays a critical role in pooling. Rather than averaging all output tokens—which would dilute the signal with padding—the mask ensures that only the hidden states of genuine tokens are aggregated. In BERT, the [CLS] token's final hidden state is used for classification, but its attention is strictly governed by the mask to summarize only the real input span, ignoring the padded tail of the batch.

Attention masks are not just for the forward pass; they are crucial for accurate loss calculation. During language modeling, the cross-entropy loss is computed only on the tokens where the mask is 1. If padding tokens were included in the loss calculation, the model would waste capacity learning to predict the [PAD] token itself. Masking the loss ensures the optimization signal focuses exclusively on reconstructing meaningful linguistic content, leading to faster convergence and better generalization.

In the Multi-Head Attention mechanism, the same attention mask is broadcast across all attention heads. The mask tensor typically has a shape of [batch_size, 1, 1, seq_len] or [batch_size, 1, seq_len, seq_len]. The singleton dimensions allow NumPy-style broadcasting to align with the [batch_size, num_heads, query_len, key_len] attention scores tensor. This ensures a consistent filtering logic across every parallel representation subspace without duplicating the mask data.