Cross-Modal Concept Bottlenecks

The core architectural modification inserts a concept bottleneck layer between the modality-specific encoders and the final classifier. This layer is a constrained latent space where each neuron corresponds to a predefined, human-specified concept. The model is forced to first predict the presence or absence of these concepts from the fused multimodal representation before the downstream task head can use them. This creates a strict information bottleneck, ensuring that all cross-modal reasoning flows through interpretable, named variables rather than opaque latent dimensions.

Training relies on a dual-objective loss function that jointly optimizes for concept accuracy and task accuracy. The bottleneck layer receives direct supervision from a dataset annotated with concept labels for each modality pair. This can be implemented in three modes:

• Independent bottleneck: Concept predictions are made solely from the bottleneck state, then passed to the task head.
• Sequential bottleneck: The task head receives both the concept predictions and the original fused representation.
• Residual bottleneck: The task head receives concept predictions plus a residual connection from the pre-bottleneck representation, allowing the model to use non-concept information when concepts are insufficient.

Concepts must be defined to span modalities coherently. For a vision-language model, a concept like 'metallic texture' must be predictable from visual features, textual descriptions, or their combination. The architecture enforces this by projecting modality-specific encodings into a shared concept embedding space before the bottleneck. This alignment ensures that the same concept neuron activates whether the evidence comes from an image patch showing a shiny surface or a text fragment describing 'brushed steel', making cross-modal reasoning directly inspectable.

The bottleneck layer functions as a test-time intervention panel. An engineer can manually set concept activations to test counterfactuals: 'What would the model predict if it believed the concept rust was present instead of absent?' This enables direct debugging of cross-modal reasoning failures. If a vision-language model misclassifies a corroded copper pipe as safe, an auditor can force-activate the 'corrosion' concept neuron and observe whether the downstream task prediction corrects itself, isolating the failure to the concept detection stage rather than the reasoning stage.

The interpretability of the entire architecture depends on the concept vocabulary — the set of human-defined concepts that populate the bottleneck. Effective vocabularies are:

• Complete: Sufficient to solve the downstream task without relying on residual connections.
• Discriminative: Concepts must distinguish between output classes.
• Unambiguous: Each concept must have a clear, consistent definition across modalities.
• Independent: Concepts should not be highly correlated, to avoid redundancy in the bottleneck. Poorly designed vocabularies lead to the model routing information through residual connections, bypassing the interpretable bottleneck entirely.

A key evaluation metric measures how responsive the final prediction is to manual concept edits. Causal intervention accuracy quantifies: if a human corrects a mispredicted concept in the bottleneck, does the downstream task prediction flip to the correct answer? High intervention effectiveness indicates that the model genuinely relies on the bottleneck concepts for reasoning. Low effectiveness reveals that the model is using non-concept shortcut features through residual connections, signaling that the concept vocabulary is incomplete or the bottleneck regularization was insufficient during training.