Glossary

Multi-Head Attention

A module that runs multiple scaled dot-product attention operations in parallel, allowing the model to jointly attend to information from different representation subspaces.

Get in touch Learn more

ML engineer managing model versions on laptop, version history visible, technical Git-like workflow.

ARCHITECTURE

What is Multi-Head Attention?

Multi-head attention is a core Transformer module that runs multiple scaled dot-product attention operations in parallel, enabling the model to jointly attend to information from different representation subspaces at different positions.

Multi-head attention is a mechanism that linearly projects the Query, Key, and Value (QKV) vectors into h distinct lower-dimensional subspaces, computing scaled dot-product attention in each head independently. This parallelization allows the model to focus on different types of relationships simultaneously—one head might track syntactic dependencies while another resolves coreference links—rather than averaging these signals into a single, noisy representation. The outputs from all heads are concatenated and linearly projected back to the model's hidden dimension.

The primary advantage over single-head attention is the ability to learn diverse relational patterns from the same input sequence. By reducing the dimension per head, the total computational cost remains roughly equivalent to a single full-dimensional attention operation, yet the model gains significantly richer representational capacity. This design is the foundational building block of the Transformer block and directly addresses the limitation of a single attention function potentially being dominated by the most prominent similarity signal.

PARALLELIZED REPRESENTATION SUBSPACES

Key Characteristics

Multi-Head Attention is the core innovation that allows Transformer models to simultaneously focus on different aspects of the input sequence. Instead of a single attention function, it projects the Query, Key, and Value vectors into multiple lower-dimensional subspaces, enabling the model to learn diverse syntactic and semantic relationships.

Parallel Attention Heads

The module runs h independent scaled dot-product attention operations in parallel. Each head operates on its own learned linear projections of the Query, Key, and Value vectors (typically with dimensions d_k = d_v = d_model / h). This allows different heads to specialize in distinct relational patterns—one head might track syntactic dependencies while another resolves coreference links.

Concatenation and Linear Projection

After computing attention in parallel, the outputs from all h heads are concatenated along the feature dimension and passed through a final linear transformation (W^O). This step synthesizes the information from the disparate representation subspaces back into a single d_model dimensional output, combining the diverse contextual signals captured by each head.

Computational Complexity

Despite running multiple attention operations, the total computational cost is comparable to single-head attention with full dimensionality. By reducing the dimension of each head (d_k = d_model / h), the matrix multiplications are cheaper. The primary bottleneck remains the quadratic complexity of computing the N x N attention score matrix, which scales with the square of the sequence length.

Joint Attention Diversity

The model jointly attends to information from different representation subspaces at different positions. This prevents the averaging effect that can occur with a single attention function. For example, in the sentence 'The animal didn't cross the street because it was too wide,' one head can focus on the syntactic subject while another attends to the adjectival modifier, resolving the ambiguity of 'it'.

Scaled Dot-Product Foundation

Each head computes attention using the formula: Attention(Q, K, V) = softmax(QK^T / √d_k)V. The scaling factor 1/√d_k prevents the dot products from growing too large in magnitude, which would push the softmax function into regions with extremely small gradients. This temperature control is critical for stable training.

Efficient Variants

Standard multi-head attention stores separate Key and Value projections for each head, which inflates the KV cache during inference. Optimized variants address this:

Multi-Query Attention (MQA): All heads share a single Key and Value projection, drastically reducing memory bandwidth.
Grouped-Query Attention (GQA): Heads are partitioned into groups sharing Key/Value projections, balancing quality and speed. Used in Meta Llama 2.

MULTI-HEAD ATTENTION EXPLAINED

Frequently Asked Questions

Clear, technically precise answers to the most common questions about the parallel attention mechanism that allows Transformer models to jointly attend to information from different representation subspaces.

Multi-head attention is a module that runs multiple scaled dot-product attention operations in parallel, allowing the model to jointly attend to information from different representation subspaces at different positions. Instead of performing a single attention function with d_model-dimensional keys, values, and queries, the model linearly projects the queries, keys, and values h times with different, learned linear projections to d_k, d_k, and d_v dimensions. Each of these projections feeds an independent attention head that computes its own attention-weighted output. The outputs of all heads are then concatenated and projected once more, yielding the final multi-head output. This design enables the model to simultaneously focus on different types of relationships—such as syntactic dependencies, coreference links, and semantic associations—within the same input sequence. The original Transformer paper used h=8 parallel attention heads, with d_k = d_v = d_model/h = 64.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

ARCHITECTURE & MECHANICS

Related Terms

Core concepts that define how multi-head attention operates and integrates within the Transformer architecture.

Scaled Dot-Product Attention

The fundamental building block of every attention head. It computes the relevance of a Query vector against all Key vectors using a dot product, scales the result by 1/√dₖ to prevent vanishing gradients in the softmax, and uses these weights to produce a weighted sum of Value vectors.

Formula: Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V
Scaling Factor: Critical for stable training when key dimensionality is large
Output: A single context vector per query position

Query, Key, Value (QKV) Projections

Each attention head learns three distinct linear transformations of the input embedding. The Query represents what the token is looking for, the Key represents what the token offers, and the Value represents the actual content to be aggregated.

Dimensionality: Typically d_model / h, where h is the number of heads
Weight Matrices: W_Q, W_K, W_V are learned independently per head
Parallelization: All projections can be computed simultaneously via matrix multiplication

Self-Attention vs. Cross-Attention

Multi-head attention operates in two distinct modes within Transformer architectures. Self-attention computes relationships within a single sequence (Q, K, V all from the same source), while cross-attention allows a decoder to attend to encoder outputs (Q from decoder, K and V from encoder).

Self-Attention: Powers contextual understanding within a sentence
Cross-Attention: Enables sequence-to-sequence tasks like translation
Causal Masking: Applied in decoder self-attention to prevent looking ahead

Multi-Query Attention (MQA)

An inference-optimized variant where all attention heads share a single set of Key and Value projection weights while retaining separate Query projections. This dramatically reduces the size of the KV cache during autoregressive decoding, cutting memory bandwidth requirements by a factor equal to the number of heads.

Memory Savings: KV cache size reduced by h×
Trade-off: Slight quality degradation vs. standard multi-head attention
Adoption: Used in PaLM and other production models

Grouped-Query Attention (GQA)

An interpolation between multi-head and multi-query attention that partitions Query heads into groups, with each group sharing a single set of Key and Value heads. This balances inference speed and model quality, achieving near multi-head quality with significantly reduced KV cache requirements.

Configuration: g groups where 1 < g < h
Adoption: Used in Llama 2 70B and subsequent models
Benefit: Near-optimal quality-efficiency Pareto frontier

Attention Head Pruning

A model compression technique that identifies and removes redundant or low-importance attention heads from a trained Transformer. Analysis often reveals that many heads learn overlapping patterns or contribute minimally to output quality.

Detection Methods: Importance scoring via gradient-based or activation-based metrics
Pruning Granularity: Individual heads can be removed without retraining
Result: 20-40% parameter reduction with minimal perplexity increase

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.