KV Cache Quantization

This defines how the quantization range is mapped.

• Symmetric Quantization: Centers the range around zero. Simpler and faster, as the zero-point is fixed, but can be inefficient if the tensor's value distribution is not symmetric.
• Asymmetric Quantization: Maps the min and max observed values to the full integer range. Utilizes the dynamic range more efficiently, especially for tensors with skewed distributions (common in activations), but adds computational overhead from a non-zero zero-point. For KV caches, symmetric quantization is often preferred for its lower runtime cost.

Quantization requires determining scale and potentially zero-point parameters. For KV caches, this is typically done via static calibration: running a small, representative dataset through the model to collect statistics (min/max or mean/std) of the key and value tensors. These statistics are used to compute fixed scaling factors applied during all subsequent inference. This avoids the runtime cost of dynamic quantization. Careful calibration is critical to minimize the impact on attention scores.

This defines the granularity of the quantization parameters.

• Per-Tensor: A single scale and zero-point is applied to an entire tensor. This is computationally simplest and most common for KV cache quantization.
• Per-Channel: Each channel (e.g., each attention head's key or value projection) gets its own scale and zero-point. This offers higher fidelity by accounting for variation across channels but increases metadata overhead and kernel complexity. It is less frequently used for caching due to these costs.

Quantization error directly injects noise into the query-key dot product, which determines attention scores. The effect is non-linear and model-dependent. Studies show that value tensors are generally more sensitive to quantization than key tensors, as errors in values propagate directly to the context vector fed into subsequent layers. Some techniques apply higher precision to values (e.g., INT8 for keys, FP16 for values) or use mixed-precision caching to mitigate this.

A KV cache is a memory buffer used in transformer-based language models to store the computed key and value tensors from previous tokens during the autoregressive decoding phase. This cache eliminates the need to recompute these tensors for every new token, which is the fundamental mechanism that makes long-context inference tractable.

• Purpose: Drastically reduces computational redundancy.
• Memory Footprint: The cache size scales linearly with batch size and sequence length, becoming the dominant memory consumer during decoding.

PagedAttention is a memory management algorithm that organizes a model's KV cache into non-contiguous, fixed-size blocks, analogous to virtual memory paging in operating systems. Popularized by the vLLM inference engine, it is a complementary technique to quantization.

• Eliminates Fragmentation: Allows flexible allocation, preventing memory waste from variable-length sequences.
• Enables Efficient Sharing: Critical for high-throughput continuous batching, as it allows physical memory blocks to be shared or swapped efficiently.

KV cache compression is an umbrella term for techniques aimed at reducing the memory footprint of the key-value cache. Quantization is a primary method within this category.

• Techniques Include: Pruning (removing low-magnitude cache entries), quantization (reducing numerical precision), and selective caching.
• Goal: To enable longer context windows or higher batch sizes within fixed GPU memory constraints, directly trading memory for capacity.

Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) are architectural modifications that reduce the size of the KV cache at the model design stage, acting as a precursor to runtime quantization.

• MQA: All query heads share a single key head and a single value head, drastically shrinking the cache.
• GQA: A tunable hybrid where groups of query heads share a key/value head, offering a better quality/efficiency trade-off.
• Synergy with Quantization: These smaller caches are then further compressed via quantization techniques.

KV cache offloading is a capacity-oriented technique that moves portions of the KV cache from GPU memory to cheaper, higher-capacity storage like CPU RAM or NVMe SSDs. It represents a different trade-off compared to quantization.

• Use Case: For extremely long contexts that exceed available GPU memory.
• Trade-off: Sacrifices latency (due to slower I/O) for vastly increased capacity. Often used in conjunction with Unified Virtual Memory (UVM) systems.

A memory-bound regime describes a computational state where system performance is limited by the speed of memory accesses rather than raw compute (FLOPs). Autoregressive decoding is typically memory-bound due to the need to read the entire KV cache for each new token.

• Quantization Impact: By reducing the precision of cached tensors, quantization directly addresses this bottleneck by decreasing the volume of data that must be transferred from memory to the compute units per token.
• Key Metric: The reduction in memory bandwidth pressure is a primary benefit of KV cache quantization.