Multimodal Tokenization

Medical images are divided into fixed-size 2D or 3D patches—such as 16x16 pixel squares from a chest X-ray or volumetric cubes from a CT scan. Each patch is flattened and linearly projected into a dense vector, creating a sequence analogous to word tokens. This patchification preserves spatial locality while converting continuous pixel data into discrete embeddings that a transformer can process. For gigapixel whole slide pathology images, hierarchical patching at multiple magnification levels captures both cellular detail and tissue architecture.

Raw genomic data—long strings of nucleotide bases (A, T, C, G)—is tokenized using k-mer decomposition, where overlapping subsequences of fixed length k are treated as discrete vocabulary items. A k=6 tokenizer produces a vocabulary of 4,096 possible hexamers. These tokens are mapped to learned embeddings that capture biological semantics. For federated multi-omics, separate tokenizers handle methylation patterns, copy number variations, and RNA-seq expression values, each with modality-specific vocabularies.

A learned modality embedding vector is added to every token to identify its source data type—imaging, text, genomic, or structured EHR. This allows a single transformer to distinguish between a patch from an MRI scan and a token from a clinical note, even when they occupy the same sequence. Without modality encoding, the model would treat all tokens as originating from a homogeneous source, losing critical context. These embeddings are analogous to positional encodings but operate on the data origin axis rather than the sequence position axis.

A shared token vocabulary spans all modalities, with reserved token ranges for each data type. Imaging patches occupy one range, genomic k-mers another, and clinical text tokens a third. Special separator tokens demarcate modality boundaries within the input sequence. This design enables cross-modal attention—where an imaging token can attend to a genomic token—while maintaining a clean separation that prevents token ID collisions. The vocabulary size typically ranges from 8,192 to 65,536 tokens depending on the number of modalities and their granularity.

Beyond standard positional encodings, multimodal medical sequences require temporal encoding for time-series data like longitudinal EHR records or sequential lab values. Each token receives both a position index within its modality segment and a global timestamp embedding. For imaging, 2D or 3D positional encodings preserve spatial relationships between patches. This dual encoding scheme ensures the transformer understands both where a token sits in its local context and when it occurred in the patient timeline.

In fragmented clinical environments, not all modalities are available for every patient. Tokenization handles this with a learned missing modality token—a special embedding that fills the sequence positions where data is absent. During training, modality dropout randomly masks entire input types to force the model to remain robust. At inference, the missing token signals to the transformer that no information exists for that modality, preventing the model from hallucinating features and enabling graceful degradation in real-world federated deployments.

A critical mechanism for distinguishing data sources within a unified sequence. A learned modality-specific vector is added to every token embedding before it enters the transformer. This allows the model to differentiate between, for example, a token originating from an EHR time-series versus a token from a genomic sequence, even when they occupy the same joint embedding space. Without modality encodings, the attention mechanism would treat all tokens as originating from a homogeneous source, losing crucial contextual provenance.

The shared, high-dimensional latent vector space into which tokens from all modalities are projected. The goal is to map semantically similar concepts—such as a radiological finding and its corresponding textual description—to nearby points in this space. This alignment enables cross-modal attention and retrieval. Training a joint space often involves contrastive objectives that pull matched pairs together and push mismatched pairs apart, creating a geometrically meaningful representation of multimodal clinical data.

An attention mechanism where the representation of one modality serves as the query to extract relevant information from the keys and values of another modality. In a federated healthcare context, genomic embeddings could attend to histopathology patch tokens to identify morphological manifestations of a specific mutation. This allows one data stream to contextually gate or guide the feature extraction of another, creating a tightly coupled, bidirectional fusion process rather than a simple concatenation of features.

A set of techniques for robust inference when one or more expected data modalities are absent at test time—a common scenario in fragmented clinical environments. Strategies include:

• Modality dropout during training to simulate missing inputs
• Learning a prior distribution in a shared latent space to generate a surrogate embedding for the missing modality
• Gated fusion mechanisms that dynamically zero out the contribution of an absent modality This ensures the model degrades gracefully rather than failing entirely.

Independent neural network branches that preprocess raw data from a single modality into a sequence of initial embeddings before fusion. For example:

• A Vision Transformer (ViT) encodes pathology images into patch tokens
• A BERT-based model encodes clinical notes into word tokens
• A graph neural network encodes protein interactions into node embeddings These encoders handle the unique inductive biases of each data type, outputting a standardized token format for the downstream fusion transformer.