Cross-Modal Attention Rollout

The core mechanism relies on recursively multiplying attention weight matrices across successive transformer layers. Starting from the final cross-modal attention layer, the rollout matrix is computed as: R = A_layer * R_prev + I, where A is the raw attention matrix and I is the identity matrix representing residual connections. This linear combination assumes that attention weights can be treated as a Markov chain, tracing how information from an earlier modality (e.g., text tokens) flows through intermediate representations to influence the final representation of another modality (e.g., image patches). The resulting rollout map aggregates the direct and indirect attention pathways into a single, interpretable heatmap.

A critical design choice in attention rollout is how to handle residual connections. The standard formulation adds an identity matrix at each step to model the skip connections that allow information to bypass the attention mechanism entirely. This is expressed as: R = 0.5 * A * R_prev + 0.5 * I. The 0.5 weighting factor normalizes the contribution between the attention pathway and the residual pathway. Without this correction, the rollout would underestimate the direct flow of information through skip connections, leading to degraded attribution maps that over-emphasize deep attention heads while ignoring the substantial information preserved via residuals.

Attention rollout can be performed in two modes:

• Class-agnostic rollout: Aggregates attention across all heads without conditioning on a specific output class, producing a general map of information flow between modalities. This reveals the model's overall grounding strategy.
• Class-specific rollout: Incorporates gradient information or the attention weights from the final classification head to weight the importance of different attention heads. This produces a map specific to a particular prediction, highlighting which cross-modal interactions were most influential for that output. The choice between modes depends on whether the goal is to understand the model's general behavior or to explain a single decision.

Unlike gradient-based attribution methods such as Integrated Gradients or Grad-CAM, attention rollout does not require backpropagation or access to model gradients. This makes it:

• Computationally cheaper: A single forward pass suffices to extract all attention matrices.
• Architecture-dependent: It relies on the assumption that attention weights are a meaningful proxy for feature importance, which is debated in the interpretability literature.
• Less precise for final-layer decisions: Gradient methods incorporate the specific output logit, while rollout provides a more holistic view of information mixing. The trade-off is between computational efficiency and the faithfulness of the explanation to the model's actual decision function.

In vision-language transformers such as ViLBERT, LXMERT, or CLIP-based architectures, cross-modal attention rollout is applied specifically to the cross-attention layers where text tokens attend to image patches and vice versa. The rollout traces how a specific word (e.g., 'dog') propagates its influence through multiple layers of cross-modal interaction to highlight the corresponding image region. The final output is a cross-modal relevance map that can be overlaid on the input image, showing which visual regions were most influenced by each textual token. This is particularly valuable for debugging vision-language grounding failures.

Attention rollout has known limitations that must be considered:

• Attention is not explanation: Raw attention weights may not correlate with feature importance, as high attention can indicate suppression or gating rather than positive contribution.
• Combinatorial explosion: In deep networks with many heads, the rollout can produce diffuse, uninterpretable maps as information disperses across all positions.
• Layer normalization effects: The rollout formula does not account for the non-linear effects of layer normalization and feed-forward networks between attention layers.
• Negative contributions: Standard rollout cannot represent inhibitory or negative cross-modal influences. These limitations have motivated extensions such as attention flow and generic attention-model explainability that incorporate gradient information to improve faithfulness.

Direct visualizations of the raw attention weights between tokens from different modalities—such as image patches and text words—at a single transformer layer. These maps reveal how a vision-language model grounds linguistic concepts in specific visual regions. Unlike rollout, which aggregates across layers, attention maps provide a layer-specific snapshot of cross-modal alignment. They are the fundamental building block upon which flow and rollout methods are constructed.

A method for tracking the propagation and aggregation of attention weights across different modalities through successive transformer layers. Flow accounts for the mixing of attention due to residual connections, providing a more faithful picture of information movement than raw attention maps. It treats attention as a flow network, computing the maximum flow of information from input tokens in one modality to output tokens in another across the entire model depth.

An extension of Gradient-weighted Class Activation Mapping that produces visual explanations for vision-language models. It computes the gradient of a text-conditioned prediction with respect to the final convolutional feature maps, highlighting the image regions most influential for that specific textual query. Unlike attention-based methods, Grad-CAM is gradient-based, capturing the sensitivity of the output rather than the attention distribution.

A causal explainability method that systematically removes or zeroes out one input modality to measure its contribution to the model's final output. By comparing predictions with and without a modality, engineers can quantify cross-modal reliance. Zeroing out text while keeping the image reveals if the model defaults to visual biases. This technique provides a direct causal estimate of modality importance, complementing correlational attention-based methods.

An attribution method that computes the path integral of gradients for all input modalities from a neutral baseline (e.g., a black image or zero embedding) to the actual input. It satisfies the completeness axiom, meaning the sum of all feature attributions equals the difference between the model's output for the input and the baseline. This provides a theoretically principled decomposition of a multimodal prediction across both visual and textual features.

Multimodal Layer-wise Relevance Propagation backpropagates a model's prediction score through the network layers, decomposing relevance and assigning it to input features across all modalities while conserving total relevance. Unlike gradient methods, LRP uses specially designed propagation rules tailored to each layer type. It produces fine-grained heatmaps showing which pixels and words jointly contributed to a specific vision-language decision.