Modality-Aware Tokenization

Each biological data type is processed by a specialized encoder before tokenization, preserving its unique statistical properties. DNA sequences use k-mer or byte-pair encoding (BPE) tokenizers that capture nucleotide context, while RNA expression vectors are discretized via vector quantization into learned codebook indices. Epigenomic tracks are segmented into continuous-valued tokens using convolutional feature extractors. This contrasts with naive approaches that force all modalities into a single tokenizer, which destroys modality-specific signal.

Despite modality-specific encoding, all tokens are mapped into a shared embedding space with a unified vocabulary. Each token receives a modality identifier prefix (e.g., [DNA], [RNA], [ATAC]) and a positional encoding that preserves genomic coordinates. This allows a single multi-modal transformer to process tokens from different sources simultaneously while maintaining awareness of each token's biological origin. The vocabulary size typically ranges from 8,192 to 65,536 tokens depending on genomic complexity.

Many omics data types are inherently continuous (e.g., gene expression counts, chromatin accessibility scores). Modality-aware tokenization employs vector quantization via learned codebooks to convert these into discrete tokens. Key approaches include:

• VQ-VAE: Encodes continuous vectors into nearest codebook entries
• Product quantization: Decomposes vectors into subspaces for efficient discretization
• Residual quantization: Iteratively quantizes reconstruction residuals for higher fidelity This discretization enables the use of cross-entropy loss during pre-training, the same objective used in language modeling.

Unlike text tokenization where sequential order suffices, genomic tokenization must preserve absolute chromosomal coordinates and strand information. Each token is augmented with:

• Chromosome identifier embedding (chr1-chr22, X, Y)
• Relative position encoding within the chromosome
• Strand directionality flag (+/-)
• Inter-token distance encoding for gapped alignments This spatial awareness is critical for tasks like variant calling and gene regulatory network reconstruction where physical proximity determines functional relationships.

The resolution of tokenization is task-dependent and configurable:

• Nucleotide-level: Single bases for variant detection (4^1 = 4 tokens)
• k-mer level: 3-6 base windows for motif discovery (4^6 = 4,096 tokens)
• Region-level: Fixed-width genomic bins (e.g., 200bp) for epigenomic tracks
• Gene-level: Whole gene tokens for expression-based analyses Multi-resolution tokenization can be combined via hierarchical tokenization where coarse tokens provide context and fine tokens capture detail, analogous to multi-scale vision transformers.

Clinical multi-omic datasets frequently have missing modalities (e.g., RNA-seq available but no methylation data). Modality-aware tokenization addresses this through:

• Learnable mask tokens: A special [MISSING] embedding that the model learns to ignore
• Modality dropout during training: Randomly omitting entire modalities to build robustness
• Conditional token generation: Using available modalities to predict missing token sequences via cross-modal attention This ensures the unified model degrades gracefully rather than failing when inputs are incomplete.

A shared lower-dimensional mathematical manifold where token embeddings from distinct biological modalities are projected and aligned. After modality-specific tokenization and encoding, RNA-seq expression vectors, ATAC-seq accessibility peaks, and DNA methylation beta values are mapped to this common space. Proximity in this space indicates semantic similarity—for example, a gene's promoter region token and its corresponding transcript abundance token should occupy nearby positions. Contrastive learning objectives enforce this alignment during pre-training.

A regularization strategy where entire data modalities are randomly masked during training. During a forward pass, all tokens from a specific omics layer (e.g., DNA methylation) are zeroed out with probability p. This forces the multi-modal transformer to learn robust representations that do not over-rely on any single data source. At inference time, the model gracefully handles missing clinical assays—a common scenario in real-world healthcare deployments where not all tests are ordered for every patient.

An architecture that explicitly models multiplicative interactions between modality-specific token embeddings by computing their outer product. Given tokenized representations from genomics, transcriptomics, and proteomics, the tensor fusion layer captures trimodal correlations that additive fusion methods miss. The resulting high-dimensional tensor is flattened and passed to downstream layers. While computationally expensive, this approach is valuable for detecting epistatic interactions where gene-gene relationships manifest only in specific proteomic contexts.

A neural network that learns token representations invariant to technical confounders while preserving biological signal. When tokenizing data from multiple sequencing centers, batch effects—systematic non-biological variation from different reagents, machines, or protocols—can dominate the learned embeddings. This autoencoder architecture uses adversarial training or conditional variational objectives to disentangle batch identity from biological state, ensuring that tokens from the same cell type cluster together regardless of which lab generated the data.