Key-Value (KV) Cache

During autoregressive decoding, a new token is generated at each step. Without a cache, the model would recompute the Key and Value vectors for all previous tokens at every step. The KV Cache stores these previously computed vectors in GPU memory.

• Process: At step t, only the Query vector for the new token is computed. This Query attends to the cached Keys and Values from steps 0 to t-1.
• Result: This transforms a quadratic O(n^2) compute operation per step into a linear O(n) one, where n is the sequence length.

The KV Cache's primary cost is memory. Its size scales linearly with the batch size, sequence length, number of layers, and head dimension.

• Formula: 2 * batch_size * seq_len * num_layers * hidden_dim * precision_bytes.
• Impact: For large models and long contexts, the cache can consume tens of gigabytes, often exceeding the memory required for the model weights themselves. This makes memory bandwidth the primary bottleneck for inference throughput.

The KV Cache's behavior is split into two distinct phases:

• Pre-fill Phase: The initial prompt is processed in parallel. The Keys and Values for all prompt tokens are computed and stored in the cache in a single, compute-bound forward pass.
• Decode Phase: Tokens are generated one by one. Each step computes a new Query and appends a single new Key-Value pair to the cache. This phase is memory-bound, limited by the speed of reading the cache from memory.

In production serving, the KV Cache enables continuous batching. Instead of waiting for all sequences in a batch to finish, a serving system can dynamically evict completed sequences and insert new requests into the batch. This requires intelligent memory management to allocate and free cache blocks without fragmentation, maximizing hardware utilization.

An architectural variant where all attention heads share a single set of Key and Value projection weights, while retaining separate Query projections. This dramatically reduces the size of the KV Cache by a factor equal to the number of heads, lowering memory bandwidth requirements during inference. MQA trades a small amount of model capacity for significant decoding speedups, making it a popular choice for production deployments where memory is the bottleneck.

An interpolation between Multi-Head Attention and Multi-Query Attention that partitions Query heads into groups. Each group shares a single set of Key and Value heads. GQA balances the inference speed of MQA with the model quality of standard multi-head attention. It achieves near-MQA memory savings for the KV Cache while recovering most of the representational power lost by aggressive sharing, as adopted in Llama 2 and subsequent models.

An IO-aware exact attention algorithm that minimizes reads and writes between GPU High Bandwidth Memory (HBM) and on-chip SRAM. By fusing the attention computation into a single tiled kernel, FlashAttention avoids materializing the full N x N attention score matrix in HBM. This reduces the memory footprint of the KV Cache during computation and accelerates both training and inference without any approximation.

The sequential generation process where a language model predicts one token at a time, appending each new token to the input sequence for the next step. Without a KV Cache, this would require re-computing the Key and Value vectors for all previous tokens at every step, resulting in quadratic complexity. The KV Cache is the primary optimization that makes autoregressive decoding computationally tractable for long sequences.

The maximum number of tokens a model can process in a single forward pass, defining the upper bound of the KV Cache size. As context windows expand from 4K to 128K or even 1M tokens, the KV Cache becomes the dominant memory consumer during inference. Managing this cache efficiently—through techniques like GQA, MQA, or KV cache quantization—is essential for serving long-context models at scale.