Missing Modality Imputation

The simplest baseline approach where the missing modality's tensor is replaced with a vector of zeros. While computationally trivial, this forces the model to treat absence as a uniform signal, often leading to degraded performance because the zero vector lies outside the learned latent distribution. It is primarily used as a lower-bound benchmark against which more sophisticated methods are measured.

Replaces the missing modality with the element-wise mean of that modality's feature vectors computed over the entire training set. This provides a statistically neutral placeholder that sits at the center of the learned distribution. While more representative than zero-imputation, it fails to capture instance-specific variation and can dilute the model's ability to make nuanced predictions for outlier cases.

Uses the available modalities to query a reference database of complete multi-modal samples. The missing modality is imputed by averaging the corresponding vectors from the K most similar retrieved cases. This method preserves patient-specific context and works well in stable clinical populations, but its reliance on a static reference set makes it brittle to distribution shifts and computationally expensive at scale.

Trains a conditional GAN where the generator synthesizes the missing modality from available inputs, and a discriminator attempts to distinguish generated from real data. The adversarial loss encourages the generator to produce representations that are indistinguishable from real measurements. This captures complex, non-linear cross-modal relationships but introduces training instability and requires careful hyperparameter tuning.

A VAE learns a shared latent distribution over all modalities during training. At inference, the available modalities are encoded to parameterize a posterior distribution in the latent space, from which the missing modality is reconstructed. This provides a probabilistic imputation with uncertainty estimates, enabling the downstream model to weigh the imputed modality's contribution appropriately rather than treating it as ground truth.

A training-time strategy rather than an inference-time fix. During multi-modal training, entire modalities are randomly zeroed out with a fixed probability. This forces the model to learn robust representations that do not over-rely on any single input source, effectively making the model natively resilient to missing modalities without requiring a separate imputation module at inference.

A generative model that learns a shared latent distribution from multiple data modalities. It is a foundational architecture for imputation because it can reconstruct a missing modality from the shared latent code inferred from the available ones.

• Product of Experts (PoE): A common MVAE variant that combines modality-specific latent distributions multiplicatively.
• Mixture of Experts (MoE): An alternative that uses additive combination for more flexible fusion.
• Application: Generating a synthetic pathology image from a patient's radiology scan and clinical notes.

A shared, high-dimensional vector space where semantically similar concepts from different modalities are mapped close to one another. Missing modality imputation often relies on projecting available data into this space to retrieve or generate the missing representation.

• Contrastive Learning: A common method for building this space by pulling paired data (e.g., an image and its report) together and pushing unpaired data apart.
• Cross-Modal Retrieval: Once built, you can query this space with one modality to find the nearest neighbor in another.

A self-supervised pre-training method that randomly masks patches of data across multiple modalities—such as pixels in an image and words in a report—and trains a model to reconstruct the missing information. This teaches the model a deep, cross-modal understanding of the data.

• Training Signal: The model learns by predicting the masked content from the unmasked context.
• Imputation by Design: The reconstruction head is inherently an imputation engine, making this a powerful pre-training strategy for missing modality tasks.

A neural network component that allows one data modality to selectively focus on the most relevant features of another modality. In imputation, cross-attention can be used to query a complete modality to generate features for a missing one.

• Query from Available: The available modality (e.g., clinical text) generates a query.
• Key/Value from Context: A learned memory bank or a generative decoder provides keys and values.
• Output: A synthetic feature vector for the missing modality (e.g., a genomic embedding) is generated through this focused, cross-modal lookup.

A model architecture where different sub-networks, or 'experts,' specialize in processing specific modalities or input types. A gating network dynamically routes information to the most relevant experts for fusion, providing inherent robustness to missing inputs.

• Sparse Activation: If a modality is missing, its corresponding expert is simply not activated.
• Dynamic Routing: The gating mechanism learns to rely on the experts for the available data, naturally handling incomplete multimodal sets without a separate imputation module.