Attention Sink

An attention sink is a direct mathematical consequence of the softmax function used in transformer attention. Softmax requires all attention scores across a sequence to sum to 1. Initial tokens, particularly the very first token, become default receptacles for "leftover" attention probability when newer, relevant tokens compete. This is not a learned feature but an emergent property of the attention computation.

The sink effect is strongest for tokens at the absolute beginning of the sequence. In models with certain positional encodings like RoPE (Rotary Positional Embedding), the initial positions maintain a stable, predictable positional signal. The model learns to allocate a baseline level of attention to these positions during training, as they are always present in its finite context window. This makes them reliable anchors.

The primary engineering utility of an attention sink is to stabilize attention distributions during extremely long generation. In the StreamingLLM framework, keeping the first few tokens (the sink) and a sliding window of recent tokens in the KV Cache prevents catastrophic attention collapse. Without the sink, when all original context tokens are evicted, the softmax can become unstable, leading to incoherent output or numerical overflow.

StreamingLLM explicitly exploits this phenomenon by mandating a fixed cache reservation for the initial tokens (e.g., the first 4 tokens). This policy ensures these sink tokens are never evicted, regardless of the total sequence length. Combined with a sliding window for recent tokens, this allows models trained on short contexts (e.g., 4K tokens) to generalize to infinite-length streams without any fine-tuning.

Attention sinks are not specific to one model architecture. They have been empirically observed in major decoder-only models like LLaMA, GPT-NeoX, and Pythia. The effect's strength can vary based on the model's positional encoding scheme and training data, but the underlying softmax constraint makes it a universal characteristic of autoregressive transformer LLMs.

It is crucial to distinguish an attention sink from semantically relevant context. A sink token receives high attention score due to its position, not its meaning. For example, a beginning-of-sequence <s> token acts as a perfect sink. Effective context window management must separate the retention of sink tokens (for numerical stability) from the retrieval of semantically relevant tokens (for task performance).

The KV Cache is a transformer optimization that stores the computed key and value tensors for previous tokens during autoregressive generation. This is the primary data structure managed by frameworks like StreamingLLM.

• Purpose: Eliminates redundant computation for tokens that remain in context, dramatically speeding up sequential token generation.
• Relation to Sink: The attention sink tokens (often the initial ones) have their KV states kept permanently in this cache, while other tokens are managed via a sliding window. Efficient cache eviction policies are needed to manage memory as new tokens are generated.

Sliding Window Attention is an efficient attention mechanism where a model's attention for a given token is restricted to a fixed window of the most recent tokens that preceded it.

• It provides a constant memory and computational cost (O(n)) for processing sequences of arbitrary length.
• In StreamingLLM, this window is applied to the majority of the sequence, while the initial attention sink tokens receive global attention. This hybrid approach maintains generation quality while enabling infinite-length inference.

Cache Eviction is the process of removing entries from a KV Cache according to a specific policy to manage memory usage.

• In standard context management, eviction happens when the token limit is reached, often using policies like Least Recently Used (LRU).
• In the StreamingLLM paradigm, eviction follows a specific rule: the sliding window moves forward, evicting tokens that fall outside the window, while the attention sink tokens are protected from eviction. This policy is key to maintaining stable attention distributions.

Positional Encoding is the method of injecting information about the order of tokens into a transformer model, which otherwise is permutation-invariant. The type of encoding influences how models handle long contexts and attention sinks.

• Absolute encodings (like in original Transformers) can degrade for positions beyond the training length.
• Relative encodings like Rotary Positional Embedding (RoPE) are more robust for length extrapolation.
• The attention sink phenomenon is observed with models using softmax attention and absolute positional cues, where initial tokens become positional anchors.

Context Window Saturation occurs when a model's fixed token limit is fully utilized, preventing the addition of new information without first removing existing context.

• Traditional solutions involve context truncation or summarization, which can lead to catastrophic information loss.
• The attention sink method directly addresses saturation by providing a stable, infinite-window alternative. Instead of hitting a hard token limit, the system continuously evicts and caches tokens from the sliding window while retaining the sink, thus avoiding performance degradation at saturation points.