Modality Fusion Entropy

Fusion entropy is computed by applying Shannon entropy to the normalized attention weights or contribution scores assigned to each modality at the model's fusion point. A uniform distribution across modalities yields maximum entropy, indicating balanced integration. A sharply peaked distribution yields low entropy, signaling that the model is relying predominantly on a single modality. The metric is formally defined as:

• H = -Σ p(m) log₂ p(m), where p(m) is the normalized importance weight for modality m
• Normalization ensures comparability across examples with different numbers of modalities
• Values range from 0 (total reliance on one modality) to log₂(M) for M modalities (perfectly uniform integration)

This property distinguishes between broad, distributed attention and narrow, concentrated attention across modalities. High dispersion (high entropy) often indicates the model is genuinely integrating complementary signals, while low dispersion (low entropy) may reveal:

• Modality neglect: The model ignores one data stream entirely, often due to training imbalances
• Shortcut learning: The model latches onto a spurious correlation in a single modality
• Redundancy: One modality contains sufficient signal, making the other superfluous for that example Tracking dispersion across a dataset reveals systematic biases in the model's fusion strategy.

Entropy is not static—it evolves across the model's depth. By measuring fusion entropy at each transformer layer in architectures with cross-modal attention, engineers can trace the integration timeline:

• Early fusion layers: Often exhibit high entropy as the model initially distributes attention broadly to gather context
• Mid-fusion layers: Entropy typically decreases as the model identifies the most relevant modality for the task
• Late fusion layers: Entropy may stabilize or drop sharply, indicating a final decision has been reached Sudden entropy collapses in specific layers can pinpoint where modality bias is introduced.

Fusion entropy is highly task-dependent. A model may exhibit balanced integration for one query type and biased integration for another. Diagnostic analysis involves:

• Stratifying entropy measurements by task category, question type, or output class
• Identifying tasks where entropy is consistently low, suggesting the model is not leveraging all available modalities
• Comparing entropy distributions between correctly and incorrectly answered examples to determine if fusion strategy correlates with accuracy This property helps isolate whether failures stem from poor unimodal encoding or defective cross-modal integration.

In production, shifts in fusion entropy can serve as an early warning system for data drift or model degradation. Monitoring entropy statistics over time enables:

• Detection of modality-specific corruption: A sudden entropy drop toward one modality may indicate sensor failure or data pipeline issues in another
• Identification of distribution shift: New input distributions may cause the model to alter its fusion strategy unexpectedly
• Triggering retraining thresholds: Sustained entropy anomalies can automatically flag the need for model recalibration This transforms fusion entropy from a static diagnostic into a continuous observability metric.

This property measures whether the entropy of attention flowing from modality A to modality B differs from the reverse direction. In vision-language models, for example:

• Text-to-image entropy quantifies how broadly the model searches the image given a text query
• Image-to-text entropy quantifies how broadly the model attends to words given an image region Asymmetric entropy patterns reveal directional biases in the fusion mechanism. A model that exhibits low text-to-image entropy but high image-to-text entropy may be overly confident in visual grounding while remaining uncertain about linguistic context.