Multi-Modal Fusion

The three primary architectural paradigms for combining modalities. Early Fusion concatenates raw data at the input level, allowing a single model to learn joint features from scratch. Late Fusion processes each modality independently with separate encoders, combining only final predictions or high-level features at the decision stage. Intermediate Fusion exchanges and combines feature representations at various internal network layers, enabling complex cross-modal interactions without the rigidity of early or late approaches.

A neural network component that allows one modality to selectively focus on the most relevant features of another. In a diagnostic context, a radiology image encoder can use cross-attention to weigh the importance of specific words in a clinical report, creating a fused representation where visual features are contextually grounded by textual semantics. This is a foundational block of the Multimodal Transformer architecture.

An architecture that explicitly models unimodal, bimodal, and trimodal interactions by computing the outer product of modality-specific feature vectors. This creates a high-dimensional tensor capturing multiplicative interactions between all feature elements. While extremely expressive, the resulting tensor is computationally expensive and often requires dimensionality reduction techniques to be practical for clinical applications.

A regularization technique where an entire data modality is randomly zeroed out or masked during training. This forces the model to learn robust representations that do not over-rely on any single input source. In production, this ensures the diagnostic system remains functional even when a clinical data stream—such as a genomic assay or a specific imaging sequence—is unavailable at inference time.

A shared, high-dimensional vector space where semantically similar concepts from different modalities are mapped close to one another. Using techniques like Contrastive Language-Image Pre-training (CLIP), an image of a malignant tumor and its textual pathological description will have a high cosine similarity. This space enables Cross-Modal Retrieval, allowing a clinician to query a genomic profile and retrieve visually similar pathology slides.

A model architecture where different sub-networks, or 'experts,' specialize in processing specific modalities or input types. A gating network dynamically routes information to the most relevant experts for fusion. For a dermatology case, the gate might heavily weight the visual expert; for a cardiac case, it might prioritize the ECG time-series expert and clinical text expert, creating a highly adaptive diagnostic system.