Query, Key, Value (QKV) Vectors

QKV vectors are not separate inputs but learned projections of the same input embedding. Each token's representation is multiplied by three distinct weight matrices—W_Q, W_K, and W_V—learned during training. This allows the model to dynamically specialize what information a token seeks (Query), what it offers (Key), and what it transmits (Value).

• Weight matrices are shared across all tokens in a sequence
• Each attention head has its own unique set of QKV projection weights
• Projection dimensions are typically smaller than the model's hidden size: d_k = d_model / num_heads

The QKV mechanism is best understood through a database retrieval analogy. The Query is what you're searching for—a specific lookup request. The Keys are the index tags on every stored item. The Values are the actual content returned.

• Query: 'Find documents about transformers'
• Keys: Titles and tags of all stored documents
• Values: The full document bodies

The attention score is the match quality between Query and each Key, and the output is a weighted sum of Values—retrieving more from highly matched items.

The Query and Key vectors must share the same dimensionality d_k to compute dot-product compatibility. The Value vector dimension d_v can differ, though it's typically set equal to d_k in standard implementations.

• Dot-product variance grows with d_k, pushing softmax into regions of extremely small gradients
• Scaling factor 1/√d_k counteracts this, keeping variance stable at 1
• Without scaling, large d_k values cause the softmax to saturate, approximating a one-hot vector and killing gradient flow

In self-attention, Q, K, and V all derive from the same sequence. In cross-attention, this symmetry breaks: Queries come from the decoder's current state, while Keys and Values come from the encoder's output.

• The decoder asks: 'What parts of the input are relevant to my next token?'
• The encoder provides the contextualized Keys and Values to answer
• This asymmetry is fundamental to sequence-to-sequence tasks like machine translation
• The encoder's output is static during decoding, enabling KV caching of the entire source sequence

Each attention head possesses its own independent set of QKV projection matrices, allowing the model to attend to different representation subspaces simultaneously. One head might focus on syntactic dependencies while another tracks long-range semantic relationships.

• With 8 heads and d_model=512, each head projects to d_k=64
• Heads operate in parallel with no weight sharing in standard multi-head attention
• Grouped-Query Attention (GQA) breaks this independence by sharing K and V projections across groups of Query heads, trading some expressivity for significant KV cache reduction

The QKV projections are the primary pathway for gradient flow through the attention mechanism. During backpropagation, gradients flow through the Value-weighted sum, then through the softmax-normalized attention scores, and finally into all three projection matrices.

• W_V receives direct gradients from the attention output
• W_Q and W_K receive gradients only through the attention weights—a more indirect path
• This asymmetry can cause training instability if one projection learns significantly faster than the others
• Proper initialization (e.g., Xavier uniform) and Layer Normalization placement are critical for balanced convergence

The mathematical core that consumes QKV vectors. Computes attention weights by taking the dot product of the Query with all Keys, scaling by 1/√d_k to prevent vanishing gradients from large dot products in high dimensions, then applying softmax to obtain a probability distribution over Values.

• Formula: Attention(Q,K,V) = softmax(QK^T/√d_k)V
• Scaling factor √d_k is critical; without it, softmax enters regions of extremely small gradients
• Produces a weighted sum where each Value's contribution is proportional to Query-Key compatibility

Runs multiple Scaled Dot-Product Attention operations in parallel, each with its own learned QKV projections. Instead of operating on d_model-dimensional vectors directly, the model projects Q, K, V into h lower-dimensional subspaces of dimension d_k = d_model/h.

• Allows the model to jointly attend to information from different representation subspaces at different positions
• Each head can learn distinct linguistic phenomena: one head may track syntactic dependencies, another coreference
• Outputs from all heads are concatenated and linearly projected back to d_model

The specific case where Queries, Keys, and Values all originate from the same input sequence. Each token generates its own QKV triplet through learned linear projections, then computes attention over all other tokens in the sequence.

• Enables every token to directly interact with every other token, regardless of distance
• Contrast with Cross-Attention, where Queries come from one sequence and Keys/Values from another
• The foundation of the Transformer's ability to capture long-range dependencies without recurrence

An inference optimization that stores previously computed Key and Value vectors for all generated tokens. During autoregressive decoding, only the new token's Query needs to be computed and scored against the cached Keys, eliminating redundant recomputation.

• Reduces per-step computation from O(n²) to O(n) for generation step n
• Memory footprint grows linearly with sequence length, becoming a bottleneck for long generations
• Variants like Multi-Query Attention and Grouped-Query Attention reduce cache size by sharing KV heads

The N × N matrix of raw compatibility scores computed as QK^T before softmax normalization. Each entry (i,j) represents how relevant token j's Key is to token i's Query.

• In Causal Attention, the upper triangle is masked to -∞ to prevent attending to future tokens
• Visualizing this matrix reveals interpretable patterns: diagonal dominance, long-range dependencies, head specialization
• The quadratic complexity bottleneck arises because this matrix scales as O(N²) with sequence length

Encodes position information by rotating Query and Key vectors in multi-head attention according to their absolute positions. The dot product between rotated Q and K naturally encodes relative position information.

• Adopted by models including Meta Llama, Mistral, and Qwen
• Rotation is applied in 2D subspaces of the embedding dimension, preserving vector norms
• Enables natural extrapolation to sequence lengths beyond those seen during training