Attention Sink

Attention sinks occur because the softmax function requires attention weights to sum to 1.0 across all tokens. When no preceding token is semantically relevant—such as at the beginning of a sequence—heads allocate their attention mass to the first few tokens as a default dumping ground. This prevents the model from distributing attention uniformly across noise, which would degrade the representational quality of the residual stream. The sink tokens effectively act as a null operation, allowing the head to abstain from meaningful information movement while satisfying the mathematical constraint of the softmax.

Empirical studies consistently show that the initial token (often the Beginning-of-Sequence token) receives attention scores 3-5x higher than semantically relevant tokens in later positions. This bias persists across all layers and heads, regardless of context length. Key characteristics include:

• The effect is robust across model scales, from small 125M parameter models to 175B+ parameter models
• Even when the first token is semantically meaningless (e.g., a period or whitespace), it still attracts massive attention
• This suggests the model learns to use position 0 as a learned constant register for discarding excess attention probability mass

Attention sinks are the key insight behind StreamingLLM, a technique that enables infinite-length text generation without cache overflow. By intentionally preserving the attention sink tokens (the first few positions) in the KV cache while evicting middle tokens using a sliding window, models can maintain stable perplexity over millions of tokens. Without preserving the sink, evicting initial tokens causes an immediate perplexity spike as attention mass is forced onto irrelevant or noisy positions. This demonstrates that attention sinks are not a bug but a functional architectural requirement.

Analysis of attention score distributions reveals a consistent two-component structure:

• Component 1: A sharp peak on the first 1-4 tokens, accounting for 40-70% of total attention mass in many heads
• Component 2: A long-tail distribution over semantically relevant tokens, following standard attention patterns This bimodal distribution is visible even when the model is processing highly structured prompts. The sink component remains remarkably stable across different inputs, suggesting it functions as a learned bias term rather than a dynamic contextual computation. Researchers have identified this as a form of implicit attention regularization.

Attention sinks interact with the residual stream in a specific way. Because the sink tokens typically have small or zero-valued information content, the weighted sum computed by the attention head adds near-zero vectors to the residual stream at the destination position. This effectively creates a no-op pathway that allows the model to bypass attention computation when no useful information exists in the context. This mechanism is complementary to the OV circuit behavior—heads with strong attention sinks often have output-value projections that map sink token representations to near-zero vectors, ensuring the sink does not inject noise.

Understanding attention sinks has direct engineering implications for KV cache management in production deployments:

• Sink-aware eviction policies that always preserve the first 4 tokens can reduce cache size by 20-30% without perplexity degradation
• Sink token compression techniques exploit the fact that sink tokens receive attention from nearly all heads, allowing them to be stored in a shared compact representation
• Quantization strategies can allocate higher precision to sink token KV entries since errors in these positions propagate to all subsequent tokens
• These optimizations are critical for high-throughput inference serving and long-context applications like document analysis and conversational agents

The primary information highway in a transformer where the attention sink phenomenon manifests. Each layer reads from and writes to this shared accumulating state. The first token's representation in this stream acts as a bias term, accumulating attention mass that is not allocated to semantically relevant tokens. This mechanism allows the model to bypass layers and preserve a resting state for excess attention probability.

The computational pathway formed by the query (Q) and key (K) projection matrices that determines attention score computation. In the attention sink phenomenon, the QK circuit learns to assign universally high scores to the initial tokens' key vectors. This creates a default resting place for attention mass, effectively acting as a learned bias term that absorbs attention not allocated to content-relevant positions.

A causal intervention technique used to isolate the functional role of attention sinks. By replacing the model's internal activation at the sink token position with a cached activation from a different forward pass, researchers can measure the causal effect on downstream predictions. This method reveals that attention sinks serve as a null operation buffer, preventing the model from attending to irrelevant tokens when no strong semantic match exists.

A memory buffer storing pre-computed key and value tensors from previous tokens during autoregressive generation. The attention sink has critical implications for KV cache management: the first few tokens' KV entries are accessed by nearly all subsequent attention heads, making them permanent cache residents. Understanding this pattern is essential for designing efficient cache eviction policies that preserve these high-access sink entries while pruning less critical tokens.

The theory that neural networks represent more independent features than they have dimensions by encoding them in almost-orthogonal directions. Attention sinks may emerge as a compressed representation of a null-attention feature, sharing the residual stream's dimensionality with other features. Sparse autoencoders can help disentangle whether the sink behavior is a dedicated monosemantic feature or an emergent property of polysemantic neuron ensembles.

A statistical framework for quantifying the contribution of specific neurons or attention patterns to model outputs. Applied to attention sinks, this analysis measures the indirect effect of the sink token through attention pathways. Results show that ablating the sink's high attention scores forces the model to distribute probability mass to semantically irrelevant tokens, degrading output quality and confirming the sink's role as an attention regularization mechanism.