Attention Weights

The foundational mechanism for computing attention weights involves three matrices: Query (Q), Key (K), and Value (V). The raw attention score between a query token and a key token is calculated as the dot product of their respective vectors. This score is then scaled down by the square root of the key dimension (√d_k) to prevent the softmax function from entering regions with extremely small gradients during training. The final attention weight is the softmax-normalized probability distribution over all keys for a given query.

• Formula: Attention(Q,K,V) = softmax(QK^T / √d_k)V
• Purpose of scaling: Mitigates vanishing gradients when d_k is large
• Result: A probability distribution where all weights sum to 1.0 for each query

Rather than computing a single set of attention weights, the transformer employs multiple attention heads in parallel. Each head projects Q, K, and V into a distinct, lower-dimensional subspace using its own learned linear projections. This allows the model to simultaneously attend to information from different representation subspaces at different positions. One head might focus on syntactic dependencies, while another captures semantic relatedness or positional proximity.

• Typical configuration: 8 to 16 heads per layer
• Head dimension: d_model / num_heads (e.g., 512 / 8 = 64)
• Output: Concatenated head outputs are linearly projected back to d_model

In decoder-only architectures (e.g., GPT models), a causal mask is applied to the attention weights before the softmax operation. This mask sets all attention scores corresponding to future tokens to negative infinity (-∞), which forces their softmax probability to zero. This ensures that the prediction for token at position i can only depend on tokens at positions less than or equal to i, preserving the autoregressive property required for text generation.

• Implementation: Upper triangular matrix of -∞ values
• Effect: Prevents information leakage from future tokens
• Contrast: Encoder models use bidirectional (unmasked) attention

In encoder-decoder transformers, cross-attention layers allow the decoder to attend to the encoder's output. Here, the queries come from the decoder's previous layer, while the keys and values come from the encoder's final output. This is the critical bridge that lets the model align source and target sequences, such as mapping an English sentence to its French translation. The attention weights in cross-attention explicitly represent the alignment between input and output tokens.

• Q source: Decoder hidden states
• K and V source: Encoder output representations
• Use case: Machine translation, sequence-to-sequence tasks

Standard self-attention has O(n²) complexity in sequence length, which becomes prohibitive for long sequences. Sparse attention patterns reduce this by restricting each query to attend only to a subset of keys. Examples include sliding window attention (local context only), dilated sliding window (periodic gaps), and global attention on special tokens like [CLS]. These patterns drastically reduce memory footprint while maintaining performance on long-document tasks.

• Longformer: Combines sliding window + global attention
• BigBird: Adds random attention to the sparse mix
• Benefit: Enables processing of sequences with 4,096+ tokens

A technique for producing visual explanations from convolutional neural networks by using the gradients of a target concept flowing into the final convolutional layer. For diagnostic imaging transformers, Grad-CAM adaptations highlight which spatial regions received the highest attention weights, enabling radiologists to verify that the model focuses on clinically relevant anatomical structures rather than spurious correlations.

An axiomatic attribution method that assigns importance scores to input features by integrating the gradients of the model's output along a straight-line path from a baseline input. This method satisfies the completeness axiom, ensuring that the sum of all feature attributions equals the difference between the prediction and the baseline—a property critical for regulatory submissions requiring faithful explanations of attention-based models.

Quantitative measures that assess how accurately an explanation method reflects the true reasoning process of the underlying machine learning model. Key metrics include:

• Sufficiency: Does removing highly-attended tokens change the prediction?
• Comprehensiveness: Does retaining only highly-attended tokens preserve the prediction?
• Correlation with attention weights: Do high-weight tokens causally influence output?

The application of feature attribution methods to identify which biological measurements most strongly influence a diagnostic model's prediction for a specific patient. By mapping attention weights onto specific genes, proteins, or metabolites, clinicians can trace a model's reasoning from input biomarker panels to output diagnoses, supporting patient-specific interpretability in precision medicine submissions.

A regulatory mechanism allowing manufacturers of AI/ML-enabled medical devices to pre-specify planned modifications without requiring a new marketing submission. When attention mechanisms are updated during model retraining, a PCCP must document how explainability methods will validate that the new attention distributions remain clinically plausible and do not introduce safety-relevant drift.