Context Window

A fixed context window is a hard, pre-trained limit (e.g., 128K tokens) where the model attends to all preceding tokens, requiring a KV cache that scales linearly with sequence length. In contrast, sliding window attention, used in models like Mistral 7B, restricts a token's attention to only the last W tokens (e.g., 4,096). This creates a constant-size KV cache, enabling linear-time complexity for infinite-length sequences but limiting long-range recall.

• Fixed Window: Full attention, linear memory O(n).
• Sliding Window: Local attention, constant memory O(1).

The context window's primary hardware constraint is the KV cache memory footprint. For a model with L layers, H attention heads, D head dimension, and precision b (e.g., 16 bits), the cache size for N tokens is: 2 * L * H * D * N * b bits. The factor of 2 accounts for separate Key and Value tensors.

For a 70B parameter model (L=80, H=64, D=128) at FP16 (b=16), a 128K context requires ~40GB of GPU memory just for the cache, placing systems squarely in a memory-bound regime where performance is limited by cache I/O bandwidth, not compute.

The context window governs two distinct computational phases:

• Prefill Phase: The entire input prompt (within the window) is processed in parallel. This phase is compute-bound and highly parallelizable, with time complexity scaling with prompt length.
• Decode Phase: Tokens are generated autoregressively. Each step attends to the entire cached context, making it memory-bandwidth bound. Latency per token is largely independent of context length, but the total memory determines the maximum possible window.

Longer contexts exponentially increase prefill latency but only linearly affect decode memory pressure.

The context window is not just a software parameter; it is defined by core model architecture choices:

• Positional Encoding: Methods like Rotary Positional Embedding (RoPE) enable efficient relative position encoding that generalizes to lengths beyond training.
• Attention Variants: Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) reduce the KV cache size per layer by having multiple query heads share single key/value heads, effectively expanding the practical context window for a given memory budget.
• Model Scaling: Larger models with more layers and heads inherently have a larger per-token cache size, making long contexts more expensive.

Several advanced techniques push the practical limits of context windows within memory constraints:

• KV Cache Quantization: Storing cache tensors in INT8 or FP8 precision can halve or quarter memory usage.
• PagedAttention: Used in vLLM, it eliminates internal fragmentation in the cache, allowing near-100% memory utilization for variable-length sequences in continuous batching.
• KV Cache Offloading: Moving less-recently-used parts of the cache to CPU RAM or NVMe storage (Unified Virtual Memory) trades I/O latency for massive context capacity.
• Cache Eviction & Compression: Policies like LRU eviction or selective pruning dynamically manage cache contents when the window is exceeded.

Choosing a context window involves balancing capability, cost, and latency:

• Throughput vs. Latency: Longer contexts reduce the number of concurrent requests (batch size) that fit in GPU memory, hurting throughput. They also increase Time To First Token (TTFT) due to longer prefill.
• Quality vs. Cost: While long contexts enable complex tasks (e.g., book analysis), the incremental utility per token often diminishes. The cost of supporting 1M tokens may be 8x that of 128K, but the quality gain is not linear.
• Hardware Selection: Long-context inference shifts the bottleneck from compute (TFLOPS) to memory bandwidth (GB/s) and capacity (GB), favoring GPUs with large, fast VRAM like the H100.

The KV cache is the memory buffer that stores computed key and value tensors from previous tokens during a transformer model's autoregressive decoding. Its size grows linearly with the number of tokens in the sequence, directly determining the maximum possible context window. Efficient management of this cache is the central challenge in long-context inference.

• Purpose: Eliminates redundant computation by reusing attention states.
• Memory Footprint: Often the largest consumer of GPU memory during inference.
• Bottleneck: Cache size and memory bandwidth limit both batch size and sequence length.

Transformer inference is split into two distinct computational phases defined by their interaction with the KV cache.

• Prefill Phase: The initial, compute-bound parallel processing of the entire input prompt. All prompt tokens are attended to simultaneously, and their key/value states are computed and written to the KV cache.
• Decode Phase: The memory-bound, token-by-token generation stage. For each new token, the model reads the entire cached context from the KV cache to compute attention, appends the new token's KV states, and predicts the next token.

The context window limit applies to the combined tokens from both phases stored in the cache.

Techniques to overcome fixed context window limits, enabling theoretically infinite generation.

• Attention Sink: A phenomenon where initial tokens (e.g., the first few) receive disproportionately high attention scores. These tokens act as stable "sinks" that anchor the attention distribution, preventing catastrophic forgetting when the cache exceeds its training length.
• StreamingLLM: A framework that leverages attention sinks to allow models trained with finite windows (e.g., 4K tokens) to generalize to infinitely long streams without retraining. It maintains a rolling cache that always keeps the sink tokens, enabling stable generation beyond the original context window.

A breakthrough memory management algorithm for the KV cache, analogous to virtual memory paging in operating systems.

• Mechanism: Organizes the KV cache into non-contiguous, fixed-size blocks or "pages." This allows:
  • Efficient Sharing: Pages can be shared between sequences in a batch (e.g., for shared prompt prefixes).
  • Zero Internal Fragmentation: Eliminates wasted memory from padding variable-length sequences.
• Impact: Popularized by the vLLM inference engine, PagedAttention enables near-optimal GPU memory utilization, allowing larger context windows and higher batch sizes within the same hardware constraints.

Attention variants designed explicitly to reduce the memory footprint of the KV cache.

• Multi-Query Attention (MQA): All query heads share a single key head and a single value head. This drastically reduces the size of the cached K and V tensors.
• Grouped-Query Attention (GQA): A generalization where queries are divided into groups, and each group shares a single key and value head. It provides a tunable trade-off between the cache efficiency of MQA and the model quality of standard Multi-Head Attention.

By shrinking the KV cache, these methods effectively expand the usable context window for a given amount of GPU memory.

Techniques to store the KV cache in a more compact form, directly increasing effective context window capacity.

• Quantization: Storing cache tensors in lower precision (e.g., FP8, INT8) instead of FP16/BF16. This reduces memory bandwidth and capacity needs, often with minimal accuracy loss.
• Compression: A broader set of methods including:
  • Pruning: Removing less important entries from the cache.
  • Selective Caching: Only caching layers or attention heads identified as critical.
• Goal: To fit longer contexts or larger batches within fixed GPU memory, pushing the practical limits of the context window.