Multi-Omics Transformer

Projects heterogeneous molecular features—gene expression counts, protein abundance values, and metabolite concentrations—into a shared, high-dimensional embedding space. Each feature is treated as a distinct token, allowing the model to process a combined sequence of multi-omics data without modality-specific encoders. This bypasses the need for separate feature extraction pipelines and enables direct cross-modal attention computation.

Leverages the core scaled dot-product attention mechanism to compute pairwise interaction weights between every token in the sequence, regardless of its omics origin. This allows a gene expression token to directly attend to a protein abundance token, capturing complex regulatory relationships such as post-translational modifications or metabolite-driven signaling. The attention matrix explicitly models inter-omics dependencies that are invisible to single-modality models.

Incorporates learnable or fixed positional encodings that can represent biological priors, such as genomic coordinates, pathway membership, or chromosomal location. This injects structural information about the molecular system into the otherwise permutation-invariant attention mechanism, allowing the model to distinguish between a mutation in a promoter region versus a distal enhancer.

Employs multiple parallel attention heads, each learning a distinct relational pattern. One head may specialize in cis-regulatory interactions within a chromosome, while another captures trans-effects across the genome. A third head might focus on protein-metabolite binding affinities. This parallel decomposition allows the model to simultaneously learn diverse biological relationship types without interference.

Introduces a dedicated learnable mask token that replaces missing omics measurements, common in real-world studies where not all assays are performed on every sample. The self-attention mechanism learns to attend away from these mask tokens or to impute their values from correlated features in other omics layers, enabling robust inference on incomplete molecular profiles without discarding partially-profiled patients.

Extracts biological insights directly from the learned attention weight matrices. High-attention edges between a genetic variant token and a protein expression token can be visualized as a putative quantitative trait locus (QTL). Aggregating attention scores across heads and layers produces a ranked list of cross-omics interactions, providing a built-in mechanism for hypothesis generation and biomarker discovery without post-hoc explainability tools.

The foundational computational block enabling the model to weigh the importance of all input tokens—regardless of their positional distance—when encoding a specific molecular feature. In a multi-omics context, this allows a gene expression token to directly attend to a protein abundance token or a methylation site token, capturing long-range cross-modal dependencies that convolutional or recurrent architectures miss. The mechanism computes three vectors—Query, Key, and Value—for each token, producing an attention score matrix that dictates information flow.

A specialized attention variant where the Query vectors are derived from one omics modality (e.g., transcriptomics) while the Key and Value vectors originate from another (e.g., proteomics). This asymmetric information flow allows the model to explicitly condition the representation of one data layer on the context of another, directly modeling regulatory relationships such as the impact of copy number variation on gene expression. It is critical for learning directed biological flows rather than simple co-occurrence.

The process of converting raw continuous omics measurements into discrete tokens suitable for a Transformer. Common strategies include:

• Value Binning: Partitioning expression values into discrete ranges.
• Gene Embedding: Treating each gene as a unique token, with its expression level added as a positional/value encoding.
• K-mer Tokenization: For raw DNA sequences, breaking the genome into overlapping nucleotide substrings. The choice of tokenization fundamentally dictates the model's vocabulary size and its ability to generalize across biological contexts.

Since the self-attention mechanism is permutation-invariant, positional encodings inject information about token order. For genomic sequences, this represents chromosomal coordinates and strand orientation. For other omics, it can encode gene proximity on a chromosome or pathway membership. Advanced implementations use learned positional embeddings or rotary position embeddings (RoPE) to capture the relative spatial and functional relationships between molecular features.

The output of the Transformer's encoder stack—a low-dimensional, unified vector representation of a biological sample that fuses information from all input omics layers. This joint latent space serves as a comprehensive molecular fingerprint for downstream tasks such as patient survival prediction, drug response classification, or biomarker discovery. The quality of this embedding determines the model's ability to generalize across different biological contexts and experimental batches.

A complementary architecture often hybridized with Transformers for multi-omics. While the Transformer models all-to-all token interactions, a GCN explicitly models known biological relationships by operating on a graph where nodes are molecular entities (genes, proteins) and edges represent interactions (protein-protein binding, regulatory links). Integrating a GCN module injects structured prior knowledge from databases like STRING or Reactome, constraining the attention mechanism to biologically plausible interactions.