Self-Attention Mechanism

Self-attention computes a weighted sum of all input tokens, where the weights are not fixed parameters but are dynamically generated from the input itself. For each token (Query), a compatibility score is calculated against every other token (Key). This allows the model to create a unique, context-dependent representation for each position, focusing sharply on relevant features regardless of their distance in the sequence.

The mechanism operates on three learned linear projections of the input:

• Query (Q): The current token asking for information.
• Key (K): A label for each token indicating what information it contains.
• Value (V): The actual information content of each token. The attention weight is the scaled dot-product similarity between Q and K, which determines how much of each V is aggregated into the output.

The core mathematical operation is Attention(Q, K, V) = softmax(QK^T / √d_k)V.

• Dot Product (QK^T): Measures raw similarity.
• Scaling (√d_k): Prevents the dot products from growing too large in magnitude, which would push the softmax function into regions of extremely small gradients, destabilizing training.
• Softmax: Converts raw scores into a probability distribution summing to 1.

Instead of performing a single attention function, the model runs multiple attention operations in parallel with different, learned linear projections. Each head can specialize in different types of relationships:

• Syntactic head: Attends to grammatical dependencies.
• Semantic head: Attends to conceptually related words.
• Long-range head: Attends to distant context. The outputs of all heads are concatenated and projected to the original dimension.

A single self-attention layer provides every token with a direct, unmediated connection to every other token in the sequence. This global receptive field is a defining advantage over Convolutional Neural Networks (CNNs), which build global context slowly through stacked local operations. In Vision Transformers, this allows a model to immediately relate a pathology in one corner of a medical image to a reference structure in the opposite corner.

The core self-attention operation is permutation-invariant—it treats the input as a set with no inherent notion of order. To make the model sensitive to spatial or sequential structure, positional encodings must be explicitly added to the input embeddings. These encodings inject information about a token's absolute or relative position, allowing the attention mechanism to distinguish 'patch A is above patch B' from 'patch B is above patch A'.

The three linear projections that form the foundation of self-attention. Each input token is projected into a query vector (what am I looking for?), a key vector (what do I contain?), and a value vector (what information do I pass on?). The attention weight between token i and token j is computed as the scaled dot-product of the query of i and the key of j, determining how much of token j's value contributes to token i's output representation.

The mathematical core defined as: Attention(Q,K,V) = softmax(QK^T / √d_k)V. The dot-product between queries and keys produces raw alignment scores. These are scaled down by 1/√d_k (where d_k is the key dimension) to prevent the softmax from entering regions of extremely small gradients when dot products grow large with high dimensionality. Without this scaling factor, gradients vanish and training becomes unstable.

Rather than computing a single attention function, multi-head attention runs h parallel attention heads on different learned linear projections of Q, K, and V. Each head can attend to different representational subspaces:

• One head may focus on short-range texture relationships
• Another may capture long-range anatomical context
• A third may detect boundary discontinuities Outputs from all heads are concatenated and linearly projected, allowing the model to jointly attend to information from different representation subspaces at different positions.

For a sequence of n tokens, self-attention computes an n × n attention matrix where entry (i,j) represents how strongly token i attends to token j. This matrix provides inherent interpretability in medical imaging—visualizing attention maps reveals which anatomical regions the model considers diagnostically relevant. In Vision Transformers processing a 16×16 patch grid, this yields a 256 × 256 attention matrix, with each row showing how one image patch relates to all others.

An IO-aware exact attention algorithm that dramatically reduces memory reads/writes between GPU high-bandwidth memory (HBM) and on-chip SRAM. Key optimizations:

• Uses tiling to compute attention in blocks that fit in SRAM
• Avoids materializing the full n×n attention matrix in HBM
• Reduces memory complexity from O(n²) to O(n)
• Achieves 2-4× speedup on long sequences while being mathematically identical to standard attention Critical for processing high-resolution medical images where patch counts are large.

Self-attention operates within a single sequence where Q, K, and V all derive from the same input—every token attends to every other token in the same modality. Cross-attention mixes information between two distinct sequences:

• Queries come from one modality (e.g., a radiology report token)
• Keys and values come from another (e.g., image patch embeddings) This mechanism enables multi-modal diagnostic fusion, where textual clinical context can direct the model's visual attention to specific anatomical regions of interest.