Scaled Dot-Product Attention

The defining feature that distinguishes this mechanism from standard dot-product attention. The dot product of Query and Key vectors grows in magnitude as the dimensionality dₖ increases, pushing the softmax function into regions of extremely small gradients. Dividing by √dₖ counteracts this effect, keeping the variance of the logits stable and preventing the softmax from saturating into a near one-hot distribution. Without this scaling, gradients vanish during backpropagation, halting learning.

The core operation requires computing an N x N attention score matrix, where N is the sequence length. This results in O(N² · d) time and O(N²) memory complexity, where d is the representation dimension. This quadratic bottleneck is the primary limitation for processing long sequences, motivating alternative architectures like sparse attention and state-space models. For a 32k token context, the raw attention matrix alone consumes over 4GB of memory.

The attention mechanism itself has no inherent notion of token order. The operation softmax(QKᵀ/√dₖ)V treats the input as a set, not a sequence. Swapping two tokens in the input simply swaps their corresponding rows in the output matrix. This property necessitates explicit positional encoding—such as sinusoidal functions or Rotary Position Embeddings (RoPE)—to inject sequence order information into the input embeddings before the attention calculation.

Unlike convolutional neural networks where filter weights are static after training, the attention weights are data-dependent. The compatibility scores between Queries and Keys are computed on-the-fly for every input sequence. This allows the model to dynamically route information based on content, enabling a single token to attend to vastly different context tokens depending on the surrounding sentence structure.

The raw attention scores are normalized using a row-wise softmax function, converting them into a probability distribution where all weights are positive and sum to 1. This ensures the output is a convex combination of Value vectors. A practical consequence is that a token's representation is always bounded by the range of the Value vectors it attends to, providing numerical stability during training.

The attention score for each Query token is computed independently of others. This allows the entire N x N matrix to be calculated in parallel across all sequence positions simultaneously, unlike recurrent neural networks which must process tokens sequentially. This massive parallelism is a key reason Transformers train efficiently on modern GPU hardware, despite the quadratic complexity.