Missing Modality Handling

A regularization strategy applied during training where entire input modalities are randomly zeroed out or masked. This forces the network to learn robust, redundant representations that do not over-rely on any single data source.

• Mechanism: Randomly drops a modality (e.g., genomic sequence) with probability p per training step
• Effect: Simulates missing data at scale, preventing co-adaptation of modality-specific branches
• Clinical relevance: Prepares the model for the fragmented reality of hospital information systems where lab results may be delayed

A generative model that learns a shared latent distribution from multiple data modalities. At inference time, the model can sample from this joint latent space conditioned only on the available modalities to reconstruct or infer the missing ones.

• Product-of-experts inference: Combines evidence from available modalities in the latent space
• Key capability: Can generate a plausible MRI scan from genomic and clinical text inputs alone
• Limitation: Requires a fully-paired training dataset to learn the joint distribution

A fusion mechanism that dynamically weights the importance of each present modality using learned attention scores. When a modality is missing, its contribution is simply set to zero, and the attention weights redistribute across the remaining inputs.

• Scalar attention: Learns a single weight per modality
• Vector attention: Computes fine-grained weights per feature dimension
• Advantage: No architectural changes needed between training and inference; naturally handles arbitrary missing patterns

A retrieval-based approach where missing modalities are imputed by finding the nearest neighbors in a shared embedding space. A patient's available data is encoded, and the closest matching complete profile is used to fill gaps.

• Cross-modal retrieval: Uses a chest X-ray embedding to retrieve the most semantically similar EHR data
• Privacy-preserving: Imputed data comes from the learned manifold, not raw patient records
• Use case: Enables cohort matching when genomic data is unavailable for a subset of patients

A tensor fusion technique that approximates the expensive outer product of modality embeddings using low-rank matrix factorization. Missing modalities are handled by inserting learned mask embeddings—fixed vectors that represent the absence of a data type.

• Mask embedding: A trainable vector substituted for any missing modality's representation
• Efficiency: Reduces fusion complexity from O(n^d) to O(n) where d is the number of modalities
• Training protocol: Mask embeddings are optimized during modality dropout to represent 'no information'

A two-stage architecture where a primary network makes an initial prediction using available modalities, and a secondary refinement network iteratively reconstructs missing features and updates the prediction. This mimics a clinician's diagnostic reasoning when test results arrive asynchronously.

• Stage 1: Coarse prediction from partial data
• Stage 2: Missing modality hallucination guided by the initial prediction hypothesis
• Benefit: Provides an interpretable intermediate output showing what the model 'imagined' the missing data to be

A regularization strategy that randomly removes entire input modalities during training to prevent the model from over-relying on any single data source.

• Forces the network to learn robust, redundant representations
• Simulates missing data scenarios at training time
• Typically applied with a fixed probability per modality (e.g., 0.2)
• Different from standard dropout which zeroes individual neurons

Example: During training, a model processing MRI, EHR, and genomics might randomly drop the genomic branch 30% of the time, ensuring the fusion module can still produce accurate predictions when genomic data is unavailable at inference.

A generative model that learns a shared latent distribution from multiple modalities, enabling reconstruction of missing data from available inputs.

• Encodes each modality into a common Gaussian latent space
• Uses a Product-of-Experts (PoE) formulation to combine modality-specific posteriors
• Can generate plausible imputations for entirely missing modalities
• Particularly effective when modalities share underlying biological structure

Clinical use case: When a patient's genomic sequencing is delayed, an MVAE trained on imaging and EHR data can infer a probabilistic genomic embedding to enable continued diagnostic workflow without waiting for lab results.

A mechanism that dynamically weights the contribution of each available modality using learned attention scores, naturally handling absent inputs.

• Computes modality-specific attention weights based on input quality and relevance
• Missing modalities receive near-zero attention, preserving prediction stability
• Enables interpretability by showing which modalities influenced a decision
• Works with cross-modal attention and self-attention variants

Implementation detail: Attention weights are often computed using a compatibility function between modality embeddings and a query vector representing the task. When a modality is missing, its key-value pairs are simply excluded from the attention computation.

A two-stage approach where missing modalities are first imputed into a shared latent space before fusion, using available modalities as conditioning inputs.

• Stage 1: Train modality-specific encoders and a shared latent space on complete data
• Stage 2: Learn cross-modal mappings to predict missing embeddings from available ones
• Supports iterative refinement for multiple missing modalities
• Can leverage contrastive learning to align embeddings across modalities

Architecture pattern: A transformer-based imputation network takes the set of available modality embeddings and predicts the missing ones by attending across the available set, then passes the completed set to a standard fusion module.

A gating mechanism that controls information flow from each modality into the shared representation, suppressing noisy or absent inputs.

• Uses learned gates (sigmoid-activated) to modulate modality contributions
• Gates can be conditioned on input features or learned embeddings
• Missing modalities are naturally gated to zero contribution
• Provides a soft, differentiable alternative to hard modality selection

Comparison to attention: While attention redistributes weight across modalities, gating independently decides how much of each modality's information passes through. This allows the model to completely suppress a modality while still attending to others.

A technique that approximates expensive tensor products in multimodal fusion using low-rank matrix factorization, providing robustness to missing modalities.

• Decomposes the full tensor fusion into modality-specific low-rank factors
• Missing modalities simply omit their factor from the computation
• Reduces parameter count from O(d^n) to O(d×r×n) where r is the rank
• Maintains expressiveness while enabling graceful degradation

Efficiency gain: For 3 modalities with 256-dimensional embeddings, full tensor fusion requires 16.7M parameters. Low-rank fusion with rank 32 reduces this to approximately 24K parameters while preserving the ability to model trimodal interactions.