Multimodal Transformer

The fundamental neural component enabling a Multimodal Transformer to fuse information. Unlike self-attention, which operates within a single sequence, cross-attention allows queries derived from one modality (e.g., a radiology image patch) to attend to keys and values from another (e.g., a clinical report token). This creates a bottleneck for information flow, where the model learns to selectively extract contextually relevant features from a source modality to enrich the representation of a target modality. It is the primary operator for learning inter-modal relationships.

A fusion strategy where cross-modal interactions occur at multiple internal layers of the transformer, rather than only at the input or output. This allows the model to learn a hierarchy of cross-modal representations, from low-level feature alignment to high-level semantic integration. For a diagnostic task, an intermediate fusion transformer might first align the texture of a CT scan with a genomic mutation marker, then later combine that fused concept with a patient's age from a clinical record. This is more computationally intensive than late fusion but captures richer, more complex synergistic relationships.

A critical regularization technique for training robust Multimodal Transformers. During training, an entire input modality (e.g., all genomic data) is randomly zeroed out or masked for a given sample. This forces the model to avoid over-relying on any single, potentially noisy, data source and to learn redundant, complementary representations. At inference time, this makes the model resilient to missing modalities, a common occurrence in real-world clinical settings where a patient's full genomic panel or a specific imaging sequence may be unavailable.

A self-supervised pre-training objective for Multimodal Transformers. The model is trained to reconstruct randomly masked patches of data across all modalities simultaneously. For example, it might predict missing pixels in an MRI slice and missing words in the accompanying radiology report. This task forces the transformer to learn a deep, joint understanding of the underlying structure of each modality and how they correlate, without requiring any manual labels. The pre-trained encoder can then be fine-tuned on smaller, labeled datasets for specific diagnostic tasks.

A transformer variant designed to handle arbitrarily large and heterogeneous multi-modal inputs without domain-specific encoders. It uses a small, fixed-size set of learned latent vectors as a computation bottleneck. These latents cross-attend to a byte-level representation of the raw inputs—whether pixel arrays, text strings, or audio waveforms—iteratively distilling relevant information. This makes it exceptionally well-suited for fusing high-dimensional modalities like whole-slide pathology images with structured clinical data, as it avoids the quadratic complexity of applying self-attention directly to the raw input.

A scaling paradigm for Multimodal Transformers where the feed-forward layers are replaced with multiple specialized sub-networks, or 'experts.' A trainable gating network learns to dynamically route each token or modality to the most relevant expert(s). This allows the model to massively increase its parameter count without a proportional increase in compute, as only a subset of experts are activated for any given input. In a diagnostic context, one expert might specialize in radiomic texture analysis while another focuses on genomic pathway integration, enabling highly specialized processing within a unified architecture.