Omics Foundation Model

Omics foundation models learn fundamental biological representations by training on millions to billions of single-cell transcriptomes or genomic sequences without manual labels. Architectures like Geneformer and scGPT employ masked token prediction—randomly masking gene expression values or sequence tokens and tasking the model with reconstructing them. This self-supervision forces the model to internalize gene-gene interaction networks, regulatory logic, and cell-type hierarchies from raw data alone. The pre-training corpus often spans tissue atlases (e.g., Human Cell Atlas), perturbation screens, and disease cohorts, creating a broad biological prior that generalizes across organisms and experimental conditions.

Unlike text models that tokenize words, omics models must convert continuous gene expression vectors or nucleotide sequences into discrete or continuous tokens. Geneformer ranks genes by expression within each cell and assigns attention-rank encoding, making the representation invariant to absolute expression magnitude. scGPT bins expression values into discrete buckets, treating each gene as a token with an associated value bin. DNA-based models like Enformer and Nucleotide Transformer use k-mer tokenization (e.g., 6-mers) or byte-pair encoding on nucleotide sequences. This modality-aware tokenization preserves biological semantics while enabling transformer architectures to process heterogeneous omics data.

The transformer's self-attention mechanism is uniquely suited to modeling biological networks. Each gene token attends to every other gene token in a cell, learning context-dependent interaction strengths that approximate regulatory relationships. In Geneformer, attention heads specialize in detecting transcription factor targets, signaling pathway members, and co-expressed modules. This is fundamentally different from static co-expression networks—the attention weights are dynamic and context-specific, varying with cell type, disease state, or perturbation. The learned attention patterns can be extracted post-hoc to perform in silico gene network reconstruction without additional training.

A hallmark of foundation models is the ability to perform tasks not explicitly seen during pre-training. Geneformer demonstrates zero-shot gene dosage sensitivity prediction—correctly ranking haploinsufficient genes without any dosage-specific training. In few-shot fine-tuning, as few as 50-100 labeled examples can adapt the model to tasks like disease classification, perturbation response prediction, or cell-type annotation. This emerges because pre-training encodes a rich manifold of cellular states where similar biological conditions cluster in the latent space. The model's embeddings serve as universal features that linear classifiers can separate with minimal supervision, dramatically reducing the need for large labeled datasets in rare disease research.

Omics foundation models exhibit emergent cross-modal transfer—a model pre-trained solely on transcriptomic data can make predictions about chromatin accessibility, protein abundance, or perturbation effects because these modalities share underlying regulatory logic. scGPT and GeneCompass demonstrate that pre-training on mouse and human data jointly produces embeddings where homologous cell types align across species. This cross-species transfer enables model organism insights to inform human biology without retraining. The key mechanism is the learning of conserved regulatory grammars—transcription factor binding motifs, co-expression patterns, and pathway structures that evolution has preserved across vertebrates.

The pre-trained model serves as a task-agnostic biological encoder that is adapted via lightweight fine-tuning heads. Common downstream tasks include:

• Cell-type annotation: Classifying cells into known types using the latent embedding
• Perturbation response prediction: Forecasting transcriptomic changes after gene knockout or drug treatment
• Gene regulatory network inference: Extracting attention weights to reconstruct TF-target relationships
• Disease state classification: Distinguishing healthy vs. diseased cells from embedding differences
• Dosage sensitivity prediction: Ranking genes by their sensitivity to copy number changes Fine-tuning typically uses parameter-efficient methods like LoRA or linear probing to preserve the pre-trained biological knowledge while adapting to specific tasks.

A context-aware, attention-based model pre-trained on a large-scale corpus of human single-cell transcriptomes. It uses a masked attention objective to learn gene network dynamics.

• Architecture: Transformer encoder stack
• Training Data: Genecorpus-30M (approx. 30 million single-cell transcriptomes)
• Key Innovation: Context-aware embeddings that shift based on the surrounding gene set, enabling in silico perturbation experiments
• Downstream Tasks: Gene dosage sensitivity prediction, chromatin dynamics, cardiac disease modeling

A generative pre-trained transformer designed for single-cell multi-omics data. It introduces a specialized attention mask and a generative pipeline that unifies gene expression, protein abundance, and chromatin accessibility.

• Architecture: Stacked transformer with multi-head attention
• Training Data: Over 33 million human cells from CELLxGENE
• Key Innovation: A unified generative pre-training objective that handles multiple omics modalities natively
• Downstream Tasks: Cell type annotation, multi-omic integration, perturbation prediction

A large-scale foundation model featuring a novel xTrimoGene architecture designed to handle the extreme sparsity and high dimensionality of single-cell expression data asymmetrically.

• Architecture: Asymmetric encoder-decoder with learnable gene embeddings
• Training Data: 50 million human single-cell transcriptomes
• Key Innovation: Tokenization strategy that treats non-zero expression values as 'words' and zero values as a separate token, dramatically reducing computational complexity
• Downstream Tasks: Drug response prediction, gene expression imputation, cell type identification

A deep convolutional and transformer hybrid model that predicts gene expression and epigenomic tracks directly from raw DNA sequence up to 200,000 base pairs away.

• Architecture: Convolutional stem with transformer attention blocks
• Training Data: Human and mouse reference genomes with paired CAGE-seq, ChIP-seq, and DNase-seq data
• Key Innovation: Dramatically extended receptive field (200 kb) compared to predecessors, capturing long-range enhancer-gene interactions
• Downstream Tasks: Variant effect prediction, regulatory element discovery, evolutionary conservation analysis

A genomic foundation model trained on entire prokaryotic genomes using a StripedHyena architecture, capable of generating biologically plausible DNA sequences and predicting molecular phenotypes.

• Architecture: Hybrid of Hyena operators and attention mechanisms
• Training Data: 300 billion nucleotides from bacterial and archaeal genomes
• Key Innovation: Operates at single-nucleotide resolution across megabase-scale contexts, enabling whole-genome generation
• Downstream Tasks: CRISPR guide RNA design, protein-DNA binding prediction, synthetic genome generation

A foundation model that adapts the BERT architecture to genomic sequences using Byte-Pair Encoding (BPE) tokenization, replacing the inefficient k-mer tokenization of its predecessor.

• Architecture: BERT-style encoder with multi-head self-attention
• Training Data: Multi-species reference genomes, including human, mouse, and zebrafish
• Key Innovation: BPE tokenization that learns a vocabulary of biologically meaningful sub-word units, dramatically improving efficiency and context length
• Downstream Tasks: Promoter prediction, transcription factor binding site identification, splice site detection

The architectural precursor to omics foundation models, adapting transformer architectures to treat genomic sequences as language. These models use self-supervised learning objectives like masked nucleotide prediction on raw DNA to learn contextual representations of genetic elements. Key examples include DNABERT and the Nucleotide Transformer, which learn to recognize promoters, splice sites, and enhancers without task-specific labels.

A shared, lower-dimensional mathematical representation where embeddings from distinct biological modalities are aligned. In omics foundation models, this space enables cross-modal reasoning—a gene's transcriptomic embedding sits near its proteomic embedding. Contrastive learning objectives pull paired profiles together while pushing unpaired ones apart, creating a unified coordinate system for biological states.

The computational process of mapping feature vectors from different biological assays into a common coordinate system. Techniques include canonical correlation analysis (CCA) and optimal transport-based methods. When properly aligned, semantically similar biological states—such as a cell in G1 phase measured via scRNA-seq and scATAC-seq—occupy proximal positions, enabling cross-modal querying and translation.

A fusion technique using cross-attention mechanisms to dynamically weigh the importance of different omics layers for specific prediction tasks. Unlike static concatenation, attention-based fusion allows the model to prioritize gene expression over methylation when predicting drug response, or invert that priority for aging-related phenotypes. This dynamic weighting is critical for handling heterogeneous biological contexts.

The strategy of converting raw biological data from different sources into discrete or continuous tokens using modality-specific encoders before feeding them into a unified transformer. DNA sequences may be tokenized as overlapping k-mers, RNA expression as binned count values, and protein abundance as continuous vectors. Each modality retains its native encoder while producing tokens compatible with the shared backbone.

A self-supervised training paradigm that pulls paired omics profiles together in the latent space while pushing unpaired profiles apart. Applied to CITE-seq data where the same cell has both RNA and protein measurements, contrastive objectives like InfoNCE loss teach the model to recognize that a cell's transcriptome and proteome are different views of the same biological entity, building robust cross-modal representations.