Grouped-Query Attention (GQA)

GQA sits precisely between Multi-Head Attention (MHA) and Multi-Query Attention (MQA) on the speed-quality curve.

• MHA (G = H): Each query head has its own unique key-value head. Highest quality, largest KV cache.
• MQA (G = 1): All query heads share a single key-value head. Fastest inference, potential quality degradation.
• GQA (1 < G < H): Query heads are partitioned into G groups. Each group shares one key-value head.

This provides a tunable knob to balance memory bandwidth and representational capacity.

The primary engineering motivation for GQA is reducing the size of the Key-Value (KV) Cache during autoregressive decoding.

• The KV cache size scales with the number of distinct key-value heads.
• For a model with H query heads and G groups, the cache is reduced by a factor of H/G compared to standard MHA.
• This directly reduces the memory bandwidth bottleneck that plagues large-batch inference, allowing for higher throughput and lower latency on memory-constrained hardware.

GQA models can be efficiently bootstrapped from existing Multi-Head Attention checkpoints through a process called uptraining.

• Mean Pooling: The key and value projection matrices for a group are initialized by mean-pooling the weight matrices of the original heads assigned to that group.
• Continued Pre-training: The converted model is trained for a small fraction (typically <10%) of the original pre-training steps.
• This allows practitioners to convert a high-quality MHA model to GQA without training from scratch, drastically reducing compute cost.

GQA has become the standard attention mechanism for large-scale production models due to its favorable inference characteristics.

• Meta Llama 2 70B: Used GQA with 8 key-value heads for 64 query heads.
• Google PaLM 2: Employed multi-query attention, the extreme endpoint of the GQA spectrum.
• Mistral 7B: Adopted GQA alongside Sliding Window Attention for efficient long-context processing.
• The technique is now a default architectural choice for models targeting deployment at scale.

Empirical evaluations demonstrate that GQA maintains model quality remarkably close to MHA while approaching the speed of MQA.

• On benchmarks like MMLU and HellaSwag, GQA models with a moderate number of groups (e.g., G=8) show negligible perplexity degradation compared to MHA baselines.
• The quality gap between MHA and MQA widens on tasks requiring fine-grained attention to multiple distinct contexts.
• GQA effectively closes this gap, recovering most of the lost representational power while retaining the memory savings.

GQA's benefits are most pronounced during high-throughput, batched inference rather than single-stream generation.

• In single-sequence decoding, the compute is dominated by matrix multiplications, and KV cache size is less critical.
• In large-batch serving (e.g., 256+ concurrent requests), memory bandwidth for reading the KV cache becomes the primary bottleneck.
• GQA reduces the total bytes read from memory per decoding step, directly translating to higher throughput and lower cost-per-token in production serving environments.

The foundational mechanism where each attention head has its own distinct Query, Key, and Value projection weights. MHA provides maximum representational capacity by allowing heads to attend to different representation subspaces simultaneously. However, this design results in a KV cache size that scales linearly with the number of heads, creating a significant memory bandwidth bottleneck during autoregressive decoding. GQA directly addresses this overhead by sharing Key and Value heads across Query groups.

An extreme optimization where all attention heads share a single set of Key and Value projection weights, while maintaining independent Query projections. MQA dramatically reduces the KV cache size and memory bandwidth requirements, leading to faster inference. The trade-off is a measurable degradation in model quality and training stability compared to MHA. GQA was developed as an interpolation between MHA and MQA to recover most of the quality loss while retaining significant speed gains.

An inference optimization that stores previously computed Key and Value vectors for all generated tokens to avoid redundant recomputation during autoregressive decoding. The memory footprint of the KV cache grows with:

• Batch size × Sequence length × Number of KV heads × Head dimension GQA reduces the number of KV heads, directly shrinking the cache size and alleviating the memory bandwidth bottleneck that often dominates inference latency in large language models.

An IO-aware exact attention algorithm that computes attention in tiles using GPU SRAM instead of repeatedly reading and writing the full attention matrix to HBM. FlashAttention is orthogonal to GQA—both can be combined for compounding speedups. GQA reduces the total data that must be moved, while FlashAttention optimizes how that data movement occurs. Together, they represent the state-of-the-art in efficient Transformer inference.

A post-training compression technique that identifies and removes redundant or low-importance attention heads to reduce computational cost. Unlike GQA, which modifies the architecture before or during training by grouping Query heads to share KV projections, pruning operates on a fully trained MHA model. GQA is a structural design choice that bakes efficiency into the architecture from the start, avoiding the potential quality loss associated with ablating heads after training.

The inherent limitation of standard self-attention where both memory and time complexity scale as O(n²) with sequence length. GQA does not address this quadratic scaling—it targets the KV cache memory bandwidth bottleneck during inference, not the computational complexity of the attention score matrix itself. For long-context efficiency, GQA is often paired with techniques like Sparse Attention or Sliding Window Attention to tackle both the memory bandwidth and quadratic compute challenges simultaneously.