KV Cache

The KV Cache eliminates redundant computation in the attention mechanism. During autoregressive decoding, each new token's attention scores are calculated against all previous tokens. Without caching, the key (K) and value (V) matrices for previous tokens would be recomputed for every new generation step, leading to O(n²) complexity. The cache stores these computed vectors, allowing the model to perform a simple lookup, reducing the computational cost of the decode stage to O(n). This is the primary source of its latency reduction.

The KV Cache transforms a compute-bound problem into a memory-bound one. The performance gain comes at the cost of significant GPU memory (VRAM) consumption.

• Memory Footprint: For a model with n_layers attention layers, n_heads per layer, and d_head dimensionality, caching K and V for a sequence of length L requires storing 2 * n_layers * n_heads * L * d_head values.
• Scaling: Memory usage scales linearly with sequence length and batch size. Long contexts or large continuous batches can exhaust available VRAM, leading to out-of-memory (OOM) errors or forced eviction/recomputation.

KV Cache efficiency directly influences key LLM latency percentiles.

• Time to First Token (TTFT): Unaffected by cache hits/misses, as this stage involves the initial prefill computation where the cache is populated.
• Inter-Token Latency: This metric benefits dramatically. Efficient cache utilization minimizes the compute per token during decoding, leading to lower and more consistent inter-token times. Cache misses or inefficient memory access patterns can cause latency spikes.
• Tail Latency (P99): Memory bandwidth saturation or contention when reading large caches can increase variance, adversely affecting P99 latency.

Continuous batching is a complementary optimization that maximizes GPU utilization by dynamically adding new requests to a running batch. It introduces complex KV Cache management:

• Padded Sequences: Requests in a batch have different sequence lengths. The cache must handle variable-length contexts, often implemented with paged attention (e.g., vLLM's PagedAttention) to avoid fragmentation.
• Cache Eviction: When a request finishes generation, its allocated cache memory must be freed for new requests. Efficient allocation and garbage collection are critical for sustained Tokens per Second (TPS).
• State Tracking: The system must maintain the correct cache state per request across potentially interrupted generation cycles.

Effective KV Cache monitoring is essential for LLM Performance Monitoring. Key indicators include:

• Cache Hit Rate: The percentage of attention operations that successfully read from cache versus requiring recomputation. A low rate indicates inefficiency.
• Cache Utilization: Percentage of allocated cache memory actively in use. Helps right-size allocations.
• Memory Pressure Metrics: GPU VRAM usage attributed to the cache, monitored via tools like Prometheus and Grafana dashboards.
• Latency Correlation: Correlate spikes in inter-token latency with cache management events (e.g., eviction, defragmentation).
• Structured Logging should record cache-related events per request for distributed tracing and root cause analysis (RCA).

Beyond basic caching, several advanced techniques and challenges exist:

• Quantized KV Cache: Storing K and V vectors in lower precision (e.g., FP8, INT8) to reduce memory footprint, at the potential cost of slight precision loss.
• Multi-Query & Grouped-Query Attention: Architectures like MQA and GQA share key/value heads across attention heads, drastically reducing the size of the KV Cache compared to standard Multi-Head Attention.
• Sliding Window Attention: Used in some models to limit cache size by only storing a fixed window of previous tokens, trading off full context for bounded memory growth.
• Cache Warming: Pre-populating the cache with a common prefix (e.g., a system prompt) for multiple requests to share, improving initial decode speed for batched requests with similar prompts.