Attention Rollout

The core mechanism involves treating attention weights as a Markov chain of information flow. Starting from the final layer, the attention matrix is recursively multiplied by the attention matrix of the preceding layer: Ã(s) = A(s) · Ã(s+1). This computation accounts for how attention in earlier layers influences attention in later layers, effectively tracing the mixing of information across all Transformer heads and layers to produce a single rollout map from output tokens back to input tokens.

Unlike Integrated Gradients or DeepLIFT, Attention Rollout does not require backpropagating gradients through the network. It operates purely on the post-hoc attention weights extracted during a forward pass. This makes it computationally lighter than gradient-based attribution methods and applicable to models where gradient access is restricted. However, this also means it ignores the value-pathway of the Transformer, assuming attention weights alone are a faithful proxy for feature importance.

A critical nuance is accounting for residual connections that skip layers in Transformer architectures. The standard rollout formulation adds an identity matrix to the attention matrix to represent the skip connection: 0.5 * A(s) + 0.5 * I. This ensures that a token's own information is partially preserved as it flows through the network, preventing the rollout signal from decaying too rapidly and providing a more stable attribution map.

Transformers use multi-head self-attention, where each head learns a distinct attention pattern. Attention Rollout must aggregate these parallel streams. The standard approach is to average the attention weights across all heads in a given layer before performing the recursive multiplication. This treats each head as an equally important information pathway, though more sophisticated variants may use head-specific weighting based on relevance scores.

The final output of Attention Rollout is a token-to-token attribution matrix where each cell (i, j) represents the cumulative attention flowing from output token i to input token j. For vision Transformers, this matrix can be reshaped into a 2D heatmap overlaid on the input image, highlighting which regions the model attended to for its classification. For text, it produces saliency-like highlighting of words most influential for a prediction.

Attention Rollout makes a strong assumption that attention weights linearly correlate with feature importance, which is not always true. It ignores the value vectors and the non-linear transformations of the feed-forward network. This can lead to misleading attributions when attention heads serve as no-op or information-gating mechanisms rather than direct feature selectors. For more faithful explanations, methods like LIME or SHAP may be preferred.

Raw attention weights show token-to-token relationships within a single layer, but they fail to capture how information flows across the entire network. Attention Rollout addresses this by recursively multiplying attention matrices across all layers, accounting for residual connections. This produces a more faithful representation of how input tokens influence the final output. Raw attention often highlights syntactic neighbors, while rollout reveals the true semantic pathways through the model.

LRP is a backpropagation-based attribution method that redistributes the model's prediction score backwards through the network using conservative propagation rules. Unlike Attention Rollout, which operates solely on attention matrices, LRP leverages both activations and weights to assign relevance scores to input features. LRP is particularly effective for convolutional networks and can be adapted to Transformers, offering finer-grained token attribution than rollout alone.

Saliency methods compute the gradient of the output with respect to the input to identify which tokens or pixels most influence a prediction. While computationally efficient, vanilla gradients can be noisy and fail to capture saturation effects. Integrated Gradients improves on this by accumulating gradients along a path from a baseline input, satisfying the completeness axiom. Attention Rollout provides a complementary structural view that is independent of gradient computation.

SHAP (SHapley Additive exPlanations) assigns each input token a Shapley value representing its marginal contribution to the prediction. For Transformers, computing exact Shapley values is computationally prohibitive, so TransformerSHAP and other approximations use attention patterns to reduce the feature space. Attention Rollout can inform these approximations by identifying the most salient token interactions before SHAP computation begins.

In encoder-decoder Transformers like T5 or BART, Attention Rollout must account for three distinct attention mechanisms: self-attention in the encoder, self-attention in the decoder, and cross-attention between them. Rollout is typically applied separately to the encoder stack to explain input processing and to the cross-attention pathway to trace how the decoder attends to encoded source tokens during generation.