Fusion Layer Probing

A diagnostic classifier (or 'probe') is a simple linear model or shallow MLP trained to predict a property of interest from the frozen hidden representations of a fusion layer. The core assumption is that if a probe can easily decode a property, that information is linearly separable and explicitly represented in the model's internal state. This technique is borrowed from BERTology in NLP and adapted for multimodal architectures.

Probes are trained to extract specific cross-modal signals from fusion layer states:

• Visual attributes from text tokens: Can a probe decode the color or shape of an object from the representation of its textual label?
• Linguistic properties from visual patches: Can a probe predict the semantic category of an image region from its fused representation?
• Alignment quality: Measuring how well the model has grounded a noun phrase to the correct bounding box region.

By placing identical probes at the output of every fusion layer, researchers trace the emergence and transformation of cross-modal knowledge through the network's depth. Typical trajectories reveal:

• Early fusion layers: Low-level feature alignment (edges, textures with simple nouns)
• Middle fusion layers: Object-level grounding and attribute binding
• Late fusion layers: High-level semantic reasoning and compositional understanding

To ensure a probe is measuring genuine multimodal fusion rather than unimodal shortcuts, researchers employ control tasks and baselines:

• Unimodal probes: Train the same classifier on representations from the vision-only or text-only encoder to quantify the information gain from fusion.
• Randomized labels: Shuffle the labels during training to establish a chance-level performance floor.
• Cross-condition probing: Train on one modality pair and test on another to assess generalization of the fused representation.

Advanced probing frameworks enforce selectivity (the probe should only extract the target property, not correlated confounds) and minimality (the probe should be as simple as possible). Techniques include:

• Amnesic probing: Training the probe while adversarially suppressing a protected attribute to isolate the target signal.
• Information-theoretic probing: Using mutual information bounds to measure how much task-relevant information is encoded, independent of probe capacity.

Correlational probing is complemented by causal interventions to verify that the decoded information is actually used by the model. After a probe identifies a fusion layer as encoding a property, researchers perform activation patching or neuron ablation at that layer and measure the downstream impact on cross-modal tasks. A drop in task performance confirms the causal role of the probed representation.

The primary tool for fusion layer probing. A simple, linear model trained on the frozen hidden states of a fusion layer to predict a specific property. If the classifier succeeds, the representation encodes that property. Common properties include part-of-speech tags, object categories, or cross-modal alignment. The classifier's accuracy serves as a proxy for the richness of the encoded information.

A complementary technique that quantifies how representations evolve. Instead of training a classifier, it compares the geometry of hidden states using metrics like Centered Kernel Alignment (CKA) or Procrustes distance. This reveals whether a fusion layer's representation is more similar to the visual input, the textual input, or a true emergent multimodal blend, without requiring labeled data.

A method for tracking how information propagates into the fusion layer. It analyzes the attention weights from the preceding cross-modal attention heads to quantify the flow of information from one modality to another. This helps answer: Is the fusion layer primarily attending to image patches or text tokens? It provides a mechanistic complement to the behavioral view of a diagnostic probe.

The practice of applying diagnostic classifiers at every layer of a multimodal model, not just the fusion layer. This creates a trajectory of how a concept emerges, is transformed, and potentially forgotten. Key insights include:

• Emergence: The layer where a concept becomes linearly separable.
• Refinement: Layers where accuracy plateaus or improves.
• Forgetting: A drop in accuracy, suggesting the model abstracts the concept away.

Moves beyond correlation to establish causality. After a probe identifies a concept in a fusion layer, this technique intervenes by zero-ablating or patching the specific neurons that encode it. If the model's output changes predictably, the encoded information is causally implicated in the model's reasoning, not just a correlated byproduct. This is the gold standard for validating probe findings.

A direct method to measure cross-modal reliance. By systematically removing or adding noise to one modality's input before the fusion layer, you can measure the impact on the resulting hidden states. A significant drop in a probe's accuracy after ablating the visual input, for example, proves the fusion layer's representation of a concept is visually grounded.