KV-Cache

During autoregressive decoding, a standard Transformer would recompute the Key (K) and Value (V) projections for the entire sequence at every step. The KV-Cache stores these tensors in GPU memory after they are first computed. When generating token t+1, the model only computes the Query (Q) for the new token and performs attention against the cached K and V from tokens 0 through t. This reduces the computational complexity of generation from quadratic to linear relative to sequence length, drastically lowering latency per step.

The memory required for the KV-Cache scales linearly with batch size, sequence length, number of layers, and hidden dimensions. The formula is: 2 * batch_size * seq_len * num_layers * hidden_dim * precision_bytes.

• FP16 Precision: Standard for inference, balancing speed and memory.
• INT8/INT4 Quantization: Applied to the cache itself to compress memory usage, often with minimal accuracy loss.
• Head Dimension: In Multi-Head Attention (MHA), the cache stores separate tensors per head, significantly increasing the total memory volume.

To reduce the size of the KV-Cache, architectural variants modify the attention heads:

• Multi-Query Attention (MQA): All Query heads share a single Key and Value head. This drastically shrinks the cache but can slightly degrade quality.
• Grouped-Query Attention (GQA): An interpolation where Query heads are divided into groups, and each group shares a single K and V head. GQA provides a balance between the memory efficiency of MQA and the quality of standard Multi-Head Attention (MHA).

A system-level optimization where the KV-Cache for a common, static prefix—such as a system prompt or few-shot examples—is computed once and reused across multiple distinct requests. When a new request arrives with the same prefix, the engine performs a hash-based lookup to retrieve the pre-computed cache blocks, bypassing the expensive prefill phase entirely. This is critical for high-throughput serving of conversational AI agents with long, complex instructions.

When the sequence length exceeds the available memory budget, the KV-Cache must be evicted. Simple sliding window attention discards the oldest tokens. More advanced strategies use attention sink theory, which observes that initial tokens often act as massive attention sinks. Eviction policies may preserve these high-attention tokens while discarding middle tokens, mitigating the 'lost in the middle' phenomenon and preserving long-context coherence without infinite memory growth.