Multimodal LIME

The core mechanism involves generating perturbed samples by independently or jointly altering input features across different modalities. For a vision-language model, this means creating variations where image segments are occluded (turned gray or black) while simultaneously removing or replacing words in the associated text. This process maps how the model's prediction changes when specific cross-modal feature combinations are present or absent, building a local dataset that captures the interaction between modalities.

A sparse linear model or a shallow decision tree is trained on the perturbed samples to act as a local, interpretable approximation of the complex multimodal model. The weights of this surrogate model directly indicate feature importance. A positive weight for a specific image segment or a text token signifies that its presence pushes the prediction toward a particular class, providing a human-readable explanation of which cross-modal features were most influential for that single prediction.

Unlike standard LIME, Multimodal LIME requires defining interpretable feature representations for each data type. For images, features are often contiguous superpixels (groups of similar pixels). For text, features are individual words or n-grams. The explanation is then presented as a weighted combination of these distinct feature types, showing, for example, that the model focused on a specific image region and a particular phrase to make its decision, explicitly revealing cross-modal grounding.

The technique prioritizes local fidelity, meaning it accurately explains the model's decision boundary in the immediate vicinity of a single input, rather than attempting to explain the entire model globally. This is crucial for debugging specific failures in multimodal systems, such as understanding why an image captioning model hallucinated an object. The explanation is faithful to that specific prediction, even if the learned local decision boundary does not generalize to other inputs.

A key strength is its complete independence from the underlying model's internal structure. Multimodal LIME treats the model as a black box, requiring only the ability to feed in perturbed inputs and observe the output probabilities. This allows it to explain any multimodal architecture—from early-fusion transformers to late-fusion dual-encoder networks—without needing access to gradients, attention weights, or internal embeddings, making it universally applicable across proprietary and open-source systems.

Perturbed samples are weighted by their proximity to the original input using an exponential kernel, ensuring that the surrogate model focuses on learning the most locally relevant behavior. However, the random perturbation process can lead to instability, where running LIME twice on the same input yields slightly different explanations. Techniques like using a larger number of samples and employing a deterministic segmentation algorithm for superpixels are critical to improving the stability and reproducibility of the cross-modal explanations.

A class of methods that assign importance scores to input features in one modality based on their interaction with and influence on features from another modality. Unlike unimodal attribution, this quantifies synergistic effects.

• Identifies which image regions are important because of a specific word
• Essential for debugging vision-language grounding failures
• Complements local surrogate models like Multimodal LIME by validating cross-modal interactions

An explainability method that systematically removes or zeroes out one input modality to measure its causal contribution to the model's final output.

• Tests if the model is over-relying on a single modality
• Reveals spurious correlations between modalities
• Often used alongside Multimodal LIME to verify that the surrogate's identified important features are causally necessary

A metric that evaluates whether the features identified as important by a multimodal explanation truly influence the model's prediction when those cross-modal features are perturbed or removed.

• Key evaluation criterion for Multimodal LIME explanations
• Measures if the local surrogate model accurately captures the black-box's decision boundary
• Low faithfulness indicates the explanation is a convincing but incorrect story

Explanations that identify the minimal, synchronized changes to inputs in multiple modalities that would alter a multimodal model's prediction to a specified alternative outcome.

• Example: 'If the dog in the image were a cat and the text said feline, the prediction would change'
• Complements Multimodal LIME by providing actionable recourse
• Used to test model robustness to coordinated cross-modal changes

An attribution method that computes the path integral of the gradients for all input modalities from a neutral baseline to the actual input, satisfying the completeness axiom.

• Guarantees that feature importances sum to the prediction difference from baseline
• Provides a gradient-based alternative to the perturbation-based Multimodal LIME
• Often used to validate LIME explanations when both methods agree on key features

The practice of training diagnostic classifiers on the hidden states of a multimodal model's fusion layers to decode what cross-modal information is represented at different stages of processing.

• Reveals where in the architecture modalities become integrated
• Helps determine which layers Multimodal LIME should target for perturbation
• Uncovers if the model learns disentangled or entangled cross-modal representations