Multimodal Data Imputation

Uses a conditional generative model trained on complete multimodal pairs to synthesize the missing modality from the available one. For example, a Multimodal Variational Autoencoder (MVAE) learns a joint latent distribution; if a genomic sequence is missing but histopathology images and EHR data are present, the model samples the latent space conditioned on the available modalities to reconstruct a plausible genomic profile. This goes beyond simple statistical mean imputation by capturing the complex, non-linear joint distribution p(X_missing | X_available).

Relies on a shared latent space where all modalities are mapped. When a modality is missing, its representation is inferred by finding the point in the joint embedding space that is most consistent with the available modalities' projections. Techniques like Contrastive Language-Image Pre-training (CLIP) can be adapted: a trained model can retrieve or generate a missing chest X-ray embedding by searching the joint space using the corresponding radiology report text, effectively performing cross-modal retrieval as an imputation strategy.

Employs cross-modal attention mechanisms where the features of the available modalities act as a query to generate the missing modality's features. A transformer can be trained to predict masked modality tokens. For instance, a Multimodal Transformer processing image patches and clinical text tokens can learn to predict the embeddings of a masked genomic sequence by attending to the relevant image regions and text descriptions, effectively hallucinating the missing data stream based on contextual cues from the other inputs.

In a Federated Learning context, a generative model can be trained across silos to produce synthetic data that fills local gaps. A central server aggregates a global generative model without seeing raw patient data. A local hospital missing genomic data for a patient cohort can use the global model, conditioned on its local imaging and EHR data, to generate synthetic genomic profiles. This preserves privacy while addressing the Missing Modality Handling problem common in fragmented clinical networks.

Training a model to be resilient to missing inputs is a form of implicit imputation. Modality Dropout randomly removes entire data streams during training, forcing the fusion network to rely on any subset of modalities. A model trained this way learns a robust conditional distribution and can naturally infer a missing modality's contribution at inference time without a separate imputation step. This builds missing modality handling directly into the predictive architecture rather than treating it as a preprocessing fix.

Treats the multimodal dataset as a high-dimensional tensor with missing entries. Low-rank factorization techniques decompose this tensor into modality-specific factors, filling gaps by assuming the data lies on a low-dimensional manifold. For example, a Tensor Fusion Network computes outer products between modality embeddings; if one modality is missing, its embedding can be solved for by completing the tensor using the known interactions between the other modalities, exploiting the multi-linear structure of the data.

The overarching discipline of designing models robust to absent data streams at inference time. Unlike imputation, which actively generates missing data, handling techniques may use modality dropout during training or learn a joint embedding space that can be queried with partial inputs. This is critical in fragmented clinical environments where a patient's genomic data might exist but their corresponding imaging is unavailable.

A generative architecture that learns a shared latent distribution from multiple modalities. The model is trained on complete datasets to encode and reconstruct all inputs jointly. At inference, an MVAE can sample from the learned joint latent space conditioned on available modalities to generate the missing ones. For example, a product-of-experts inference network can combine partial evidence from an EHR and a chest X-ray to synthesize a plausible genomic embedding.

An attention mechanism where the representation of one modality guides the feature extraction of another. In imputation, a cross-modal attention layer can use the contextual embeddings from a clinical text note to attend to and fill gaps in a structured lab values matrix. This allows one data stream to contextually inform the reconstruction of a second, rather than relying on a static prior.

A privacy-preserving strategy where clients share abstract class prototypes—representative embeddings of each category—instead of raw gradients. For imputation, sites can share modality-specific prototypes of complete patient representations. A local model can then use these shared prototypes as conditional priors to guide the generation of missing data without ever exposing the original training samples.

A shared latent vector space where representations of different modalities are mapped to enable direct comparison. A well-aligned joint embedding space is the foundation of cross-modal imputation. If a model maps both histopathology patches and genomic sequences to the same space, the coordinates of an available image can be used to retrieve or generate the nearest plausible genomic vector, effectively filling the missing modality.

A regularization strategy that randomly drops entire input modalities during training. This forces the model to learn robust representations that do not over-rely on any single data source. By simulating missing data scenarios, modality dropout trains the fusion mechanism to perform implicit imputation, learning to reconstruct the semantic content of a dropped modality from the remaining available streams.