Cross-Modal Representation Similarity

The gold-standard metric for measuring similarity between two sets of representations, even if they have different dimensionalities. CKA computes the Hilbert-Schmidt Independence Criterion (HSIC) between centered kernel matrices.

• Invariant to orthogonal transformations and isotropic scaling
• Captures non-linear relationships using RBF kernels
• Range: 0 (no similarity) to 1 (identical structure)
• Used to compare vision and language encoder outputs before fusion

A method that first applies singular value decomposition to filter out noise dimensions, then performs canonical correlation analysis to find maximally correlated directions between two representation spaces.

• Denoising step removes low-variance neurons
• Returns a similarity score and the aligned subspaces
• Effective for comparing layer-wise representations across modalities
• Reveals at which depth cross-modal alignment emerges

A framework that abstracts away from individual neurons by comparing the pairwise distance structure within each modality's representation space. An RSM is an N×N matrix where entry (i,j) is the dissimilarity between stimuli i and j.

• Second-order isomorphism: compares geometry, not coordinates
• Enables cross-modal comparison via correlation of RSMs
• Reveals whether text and image encoders preserve similar relational structures
• Agnostic to the dimensionality of each modality

Quantifies the statistical dependence between representations from different modalities by estimating how much knowing one representation reduces uncertainty about the other.

• Uses contrastive lower bounds like InfoNCE for estimation
• Directly linked to the contrastive learning objective in CLIP-style models
• High mutual information indicates strong cross-modal alignment
• Sensitive to both linear and non-linear dependencies

Measures the minimum linear transformation error required to map one modality's representation space onto another. Finds the optimal orthogonal rotation, translation, and scaling.

• Closed-form solution via singular value decomposition
• Returns a residual distance after optimal alignment
• Tests whether cross-modal representations are linearly mappable
• Low distance suggests the model has learned a unified representational geometry

Tracks how the similarity between modality representations evolves across the depth of a multimodal transformer. Early layers typically show low similarity, while fusion layers exhibit high alignment.

• Plotted as a similarity curve from input to output
• Identifies the emergence point where modalities converge
• Diagnoses whether fusion happens too early or too late
• Reveals architectural bottlenecks in cross-modal processing

The primary quantitative metric used to measure similarity between two representation matrices, often from different modalities. CKA is invariant to orthogonal transformations and isotropic scaling, making it ideal for comparing neural network layers. Key properties:

• Uses a kernel function (typically linear or RBF) to compute similarity
• Produces a score between 0 (dissimilar) and 1 (identical)
• Robust to the dimensionality mismatch common in cross-modal comparisons
• Widely adopted after Kornblith et al. (2019) demonstrated its reliability over other metrics like CCA

The practice of training diagnostic classifiers on the hidden states of a multimodal model's fusion layers to decode what cross-modal information is represented at different stages of processing. Probing workflow:

• Extract hidden states from a specific fusion layer for a batch of multimodal inputs
• Train a simple linear classifier to predict a property of interest (e.g., object category, spatial relationship)
• High probing accuracy indicates that the representation encodes that property
• Comparing probing accuracy across layers reveals the progressive refinement of cross-modal alignment through the network depth

An explainability method that systematically removes or zeroes out one input modality to measure its causal contribution to the model's final output and assess cross-modal reliance. Ablation strategies:

• Zero-ablation: Replace one modality's input with zeros and measure the prediction change
• Noise-ablation: Replace with Gaussian noise to test robustness
• Mean-ablation: Replace with the dataset's average embedding for that modality
• A large drop in accuracy after ablating a modality indicates high reliance; minimal change suggests the model can compensate with the remaining modality

A method for tracking the propagation and aggregation of attention weights across different modalities through the successive layers of a transformer-based multimodal architecture. How it works:

• Start with the raw attention matrices from each cross-modal attention head
• Apply attention rollout to linearly combine matrices across layers, accounting for residual connections
• The result is a single flow map showing how information from, say, text tokens influences image patch representations at each processing stage
• This reveals whether the model grounds specific words in specific visual regions early or late in the network

A metric that evaluates whether the features identified as important by a multimodal explanation truly influence the model's prediction when those cross-modal features are perturbed or removed. Evaluation protocol:

• Generate an explanation identifying the most important features across modalities
• Perturb or remove those features and measure the prediction change
• A faithful explanation produces a large, monotonic drop in confidence as important features are removed
• An unfaithful explanation identifies features that, when removed, cause little or erratic change, indicating the explanation does not reflect the model's true decision process