Modality Importance Weighting

Produces a single, interpretable weight per modality (e.g., text: 0.7, image: 0.3) that sums to 1.0. This global attribution answers 'Which sense did the model use most?' without requiring per-feature granularity. The score is derived by aggregating cross-modal attention weights or gradient magnitudes across all layers and tokens for a given modality.

Computes the norm of the gradient of the model's output with respect to each modality's input features. A higher gradient magnitude indicates higher sensitivity. This method satisfies the completeness axiom when integrated along a path, ensuring the sum of modality weights equals the total prediction score difference from a baseline.

In transformer-based multimodal models, modality importance is computed by tracking the cross-modal attention flow from the [CLS] token back to each modality's input tokens. The cumulative attention mass received by text tokens versus image patches provides a direct, parameter-free importance weight that reflects the model's internal information routing.

Systematically zeroes out or replaces one modality with a neutral baseline (e.g., a blank image or zeroed text embedding) and measures the drop in prediction confidence. The magnitude of the probability delta directly quantifies that modality's causal contribution. This is a model-agnostic, black-box approach requiring no access to internal weights.

Treats each modality as a cooperative game player and computes its marginal contribution averaged over all possible modality subsets. This game-theoretic approach guarantees a fair, axiomatically justified distribution of credit. It accounts for complex cross-modal interactions where the value of one modality depends on the presence of another.

Modality importance weights serve as a critical diagnostic tool for detecting unintended modality bias. If a visual question-answering model assigns 0.95 weight to text and 0.05 to images, it reveals a language-prior shortcut. Engineers use this signal to apply targeted regularization or modality dropout during training to enforce balanced cross-modal reasoning.

An explainability method that systematically removes or zeroes out one input modality to measure its causal contribution to the model's final output. Unlike correlation-based importance weighting, ablation establishes a counterfactual: what would the model predict if it couldn't see the image? The drop in confidence directly quantifies reliance on the ablated modality.

The application of Shapley Additive Explanations to fairly distribute the credit for a multimodal model's prediction among input features from all contributing modalities. This game-theoretic approach satisfies key axioms like consistency and local accuracy, producing a modality-level importance weight that is mathematically guaranteed to sum to the prediction difference from a baseline.

An attribution method that computes the path integral of gradients for all input modalities from a neutral baseline to the actual input. By satisfying the completeness axiom, it ensures that the sum of feature attributions across modalities exactly equals the prediction. This provides a principled decomposition of the output into per-modality contributions.

A diagnostic metric that measures the uncertainty in how a model distributes its attention across different modalities at the fusion point. High entropy indicates a balanced integration strategy, while low entropy signals heavy bias toward one modality. This complements importance weighting by characterizing the distribution shape rather than just a point estimate of contribution.

A technique from causal inference applied to multimodal models to identify specific cross-modal neurons that function as causal mediators between an input concept and the model's output. While importance weighting quantifies overall modality contribution, mediation analysis pinpoints the exact computational pathways through which one modality influences the final prediction.