Multimodal Layer-wise Relevance Propagation (MLRP)

The foundational single-modality technique upon which MLRP is built. LRP operates under a conservation principle, meaning the total relevance score at the output is fully redistributed backward through the network layers to the input features without loss or gain.

• Deep Taylor Decomposition: A core LRP rule that approximates neuron activations with a Taylor expansion to assign relevance.
• Alpha-Beta Rule: A parameterized propagation rule that controls the ratio of positive to negative relevance flow.
• Relevance Conservation: The sum of relevance scores at any layer equals the model's output score, ensuring a complete accounting of the prediction.

A class of methods that assign importance scores to input features in one modality based on their interaction with and influence on features from another modality. MLRP is a specific implementation of this broader concept.

• Interaction Attribution: Quantifies how a specific image region and a text token jointly influence the prediction beyond their individual contributions.
• Bi-modal Heatmaps: Visual outputs that overlay textual relevance scores directly onto corresponding image regions.
• Modality-Specific Propagation: Requires distinct propagation rules tailored to the mathematical operations of each modality's encoder.

A critical evaluation metric that measures whether the features identified as important by an explanation technique like MLRP truly influence the model's prediction. A faithful explanation accurately reflects the model's internal reasoning, not just plausible-sounding correlations.

• Perturbation Testing: Systematically removing or altering the highest-attributed features and measuring the resulting drop in prediction confidence.
• Completeness Check: Verifying that the sum of all relevance scores across modalities equals the final output logit.
• Contrastive Evaluation: Comparing MLRP attributions against random baselines to ensure the identified cross-modal interactions are statistically significant.

An alternative attribution method that computes the path integral of gradients from a neutral baseline input to the actual multimodal input. Unlike MLRP's layer-by-layer backpropagation, this method satisfies the completeness axiom by design.

• Baseline Selection: Requires defining a neutral reference point, such as a black image and a zero-embedding text, from which to accumulate gradients.
• Path Linearity: Assumes a straight-line path in the input space, which may not capture the non-linear cross-modal interactions that MLRP's deep redistribution can reveal.
• Computational Cost: Typically requires multiple forward and backward passes to approximate the integral, making it more expensive than a single MLRP pass.

The practice of training diagnostic classifiers on the hidden states of a multimodal model's fusion layers to decode what cross-modal information is represented at different processing stages. This complements MLRP by validating the semantic meaning of the representations that receive high relevance.

• Probe Training: A lightweight linear classifier is trained to predict a specific property, like a bounding box coordinate, from a frozen layer's hidden state.
• Layer-wise Analysis: Probing across layers reveals how cross-modal grounding emerges progressively, from simple co-occurrence to complex semantic alignment.
• Relevance Correlation: High MLRP relevance for a neuron that also scores highly on a concept probe provides strong evidence of a causal, interpretable role.

An explainability method that systematically removes or zeroes out one input modality to measure its causal contribution to the model's final output. This provides a coarse-grained complement to MLRP's fine-grained feature attribution.

• Zero-Ablation: Replacing an entire modality with zeros and observing the prediction change.
• Noise-Ablation: Replacing a modality with Gaussian noise to test robustness to uninformative inputs.
• Cross-Modal Reliance Diagnosis: A large prediction drop after ablating one modality indicates heavy reliance, which MLRP can then decompose into specific feature-level interactions.