Modality Encoding

Each modality is assigned a unique, learnable embedding vector that is summed with the token or patch embeddings at the input layer. During training, backpropagation optimizes these vectors so the model learns to differentiate between, for example, genomic sequences and radiology images. This is analogous to positional encodings but operates on the type axis rather than the sequence axis.

By adding a modality-specific vector to otherwise identical token embeddings, all data streams can be processed by a single shared transformer. The self-attention mechanism uses the modality signal to learn distinct attention patterns for each data type. This eliminates the need for separate encoders and enables cross-modal attention to occur naturally in deeper layers.

The set of modality encodings forms a type vocabulary that can be extended as new data sources are added. Common clinical modalities include:

• Imaging (CT, MRI, X-ray)
• Structured EHR (lab values, vitals)
• Genomic sequences (DNA, RNA)
• Clinical notes (free text) Each receives a distinct vector, allowing the model to scale to heterogeneous hospital data.

Without modality encodings, a transformer would treat a patch from a histopathology slide identically to a token from a radiology report, losing critical context. The encoding acts as a provenance tag that persists through every layer, enabling the model to apply modality-appropriate transformations while still allowing cross-modal interactions in the shared latent space.

In federated multi-modal fusion, modality encodings are particularly valuable because they are compact and do not expose raw data. A hospital can share its modality embedding definitions without revealing patient information. The global server can aggregate these type vectors alongside model weights, enabling a consistent modality vocabulary across the entire decentralized network.

When a modality is absent at inference time—a common occurrence in fragmented clinical environments—the model can use a learned null embedding or simply omit that modality's tokens. Because the encoding is additive, the remaining modalities retain their identity signals. This design supports modality dropout during training, forcing the network to avoid over-reliance on any single data source.

Architectures that process multiple data types simultaneously by treating inputs as distinct token sequences. Modality encoding is the critical mechanism that allows these models to differentiate between, for example, a patch from a histopathology slide and a token from a clinical note, enabling unified attention computation across all sources.

An attention mechanism where the representation of one modality guides the feature extraction of another. Modality encodings ensure the model maintains source awareness during this process, allowing genomic embeddings to contextually inform the interpretation of radiology images without losing track of which data originated where.

Independent neural network branches designed to extract salient features from a single data type before fusion. Each encoder's output is tagged with a learned modality encoding vector, which serves as a provenance marker when the embeddings are concatenated and passed to the shared fusion module.

A regularization strategy that randomly drops entire input modalities during training. This forces the model to rely on modality encodings to dynamically adapt to missing inputs, preventing over-dependence on any single data source and building robustness for fragmented clinical environments where certain tests may be unavailable.

A shared latent vector space where representations of different data types are mapped to enable direct comparison. Modality encodings prevent the collapse of distinct modalities into indistinguishable points, preserving the unique semantic structure of each source while allowing cross-modal retrieval between medical images and text reports.

A mechanism that dynamically weights the importance of different input modalities using attention scores. The modality encoding provides the transformer with the necessary signal to compute these weights, allowing the model to focus on the most clinically relevant data source—such as prioritizing lab results over demographic data—for a specific diagnostic prediction.