Multimodal Masked Autoencoder

A core innovation is the application of a joint masking pattern across modalities. For a paired image-text sample, a high percentage of image patches (e.g., 75%) and text tokens (e.g., 50%) are randomly masked. The model is forced to predict the missing pixels and words using the unmasked complementary modality as context, learning a cross-modal grounding that a single-modality MAE cannot achieve.

M-MAEs inherit the efficient asymmetric architecture from vision MAEs. The encoder processes only the small subset of visible, unmasked patches from all modalities, drastically reducing pre-training compute. A lightweight, shared or modality-specific decoder is then tasked with reconstructing the masked content. This design ensures scalability to large-scale multi-modal datasets without prohibitive resource costs.

The training loss is a composite function that drives representation learning:

• Pixel Reconstruction Loss: Mean Squared Error (MSE) between the original and reconstructed masked image patches.
• Text Token Reconstruction Loss: Cross-entropy loss for predicting the correct vocabulary IDs of masked words.
• Cross-Modal Contrastive Loss (Optional): Aligns the global representations of the paired modalities in a joint embedding space, further enhancing semantic coherence.

To process fundamentally different data types, M-MAEs employ a modality-agnostic input projection. An image is first tokenized into non-overlapping patches and linearly projected into a sequence of vectors. Text is tokenized into sub-word units and embedded. These sequences are concatenated, added with positional and modality-type embeddings, and fed into a single, shared Vision Transformer (ViT) backbone, treating all data as a unified token stream.

By training with high masking ratios, M-MAEs inherently develop robustness to missing data at inference time. The model learns to infer missing information from any available modality. For example, if a clinical report is unavailable, the model can still generate a plausible report or complete its own masked image patches based solely on the visual evidence, making it ideal for real-world clinical settings with incomplete patient records.

The pre-trained encoder serves as a powerful feature extractor for multi-modal diagnostic tasks. After pre-training, the decoder is discarded, and a task-specific head is attached. The model can be fine-tuned on small, labeled datasets for tasks like:

• Visual Question Answering (VQA) on radiology images.
• Automated report generation from chest X-rays.
• Zero-shot cross-modal retrieval between imaging and genomic data.

The foundational learning paradigm where a model generates its own supervisory signal from unlabeled data. In a Multimodal Masked Autoencoder, the objective is to reconstruct randomly masked patches of an image and corresponding tokens in a clinical report. This forces the model to learn a robust, cross-modal joint representation without requiring expensive manual annotation.

The standard backbone for processing visual data in this architecture. An image is divided into non-overlapping patches, linearly embedded, and processed by a transformer. Crucially, the encoder only processes the unmasked patches, drastically reducing compute. This sparse computation is key to the efficiency of masked autoencoders on large volumetric scans like CT and MRI.

The model's objective goes beyond reconstructing raw pixels. Targets often include:

• Pixel reconstruction: Regenerating masked image regions.
• Text token prediction: Rebuilding masked words in a radiology report.
• Contrastive alignment: Pulling the global embedding of the image close to its corresponding report in a joint embedding space. This multi-objective training creates a holistic understanding of the patient case.

A critical technique to prevent the model from over-relying on a single data stream. During training, an entire modality—such as the clinical text—is randomly masked or zeroed out. This forces the model to learn robust, independent representations from each modality and handle missing modality imputation gracefully at inference time, a common scenario in real-world clinical settings.

A defining architectural trait. The encoder is a large, heavy Vision Transformer that operates only on visible patches. The decoder is a small, lightweight transformer that reconstructs the full input from the combination of encoded visible patches and learned mask tokens. This asymmetry ensures pre-training is computationally efficient while the encoder remains powerful for downstream transfer learning.

After pre-training on large, unlabeled multimodal datasets, the encoder is transferred to supervised tasks with limited labeled data. Applications include:

• Radiogenomics: Predicting genetic mutations from CT scans.
• Structured report generation: Automating clinical documentation.
• Prognostic index creation: Fusing imaging and lab data to predict survival outcomes. The pre-trained weights provide a massive performance boost over training from scratch.