Single-Cell Foundation Model

Models like Geneformer and scGPT are pretrained on tens of millions of single-cell transcriptomes using self-supervised objectives such as masked gene token prediction or next token prediction. This phase ingests data from diverse tissues, species, and experimental conditions to learn a universal representation of gene-gene interaction dynamics. The pretraining corpus often exceeds 30 million cells, spanning hundreds of studies from resources like the Human Cell Atlas. The model learns context-dependent gene relationships without requiring labeled data, capturing regulatory logic and cellular states in its attention weights.

Unlike static gene embeddings, foundation models generate context-aware representations where the same gene receives different vector embeddings depending on its surrounding transcriptomic neighborhood. This mirrors the attention mechanism in natural language processing: a gene's representation is influenced by the co-expression patterns of other genes in the same cell. This allows the model to distinguish between a gene's role in different cell types or disease states, enabling fine-grained biological reasoning without retraining.

A defining capability of foundation models is zero-shot generalization. After pretraining, models like Geneformer can perform tasks they were never explicitly trained on—such as predicting disease gene candidates or dosage sensitivity—by simply presenting the task as an in silico perturbation experiment. The model compares embeddings before and after gene deletion or overexpression in its attention context, quantifying the impact. This eliminates the need for task-specific architectures and labeled training data for every new biological question.

Pretrained models serve as a foundation that can be rapidly adapted to specific tasks through parameter-efficient fine-tuning. Key applications include:

• Cell type annotation: Mapping query cells to reference atlases
• Gene regulatory network inference: Identifying transcription factor targets
• Perturbation prediction: Forecasting transcriptional response to genetic or chemical perturbations
• Batch integration: Harmonizing data across experiments without explicit correction algorithms
• Disease state classification: Distinguishing healthy from pathological cell states

The transformer attention mechanism provides a built-in interpretability layer. By extracting attention weights between gene tokens, researchers can identify which gene-gene interactions most influenced a prediction. This reveals gene regulatory relationships and signaling pathways directly from the model's internal logic. For example, in Geneformer, attention heads can be analyzed to discover transcription factor hierarchies and co-regulated gene modules, offering mechanistic insights alongside predictions.

Emerging foundation models extend beyond transcriptomics to integrate multi-omic data including chromatin accessibility (scATAC-seq), surface proteins (CITE-seq), and spatial coordinates. Architectures like scGPT support multiple token types within a unified framework. Additionally, models pretrained on human data can transfer to cross-species analysis, leveraging conserved gene orthologs to study model organisms without species-specific retraining. This universality positions foundation models as a computational microscope applicable across biological domains.

The computational alignment of multiple single-cell datasets from different conditions, technologies, or donors into a shared latent space. Foundation models require massive, diverse training corpora, making robust integration methods like Harmony and scVI critical preprocessing steps. These algorithms correct for technical batch effects while preserving genuine biological variation, ensuring the model learns universal rather than dataset-specific patterns.

The fusion of multiple single-cell data modalities—transcriptomics, epigenomics, and proteomics—into a unified representation. Advanced foundation models like scGPT accept multi-modal inputs, learning joint embeddings that capture holistic cellular states. Key methods enabling this include:

• Seurat WNN: Learns cell-specific modality weights
• MOFA+: Factor analysis for multi-omics
• TotalVI: Variational inference for CITE-seq data

A supervised machine learning approach that projects cell-type annotations from a well-characterized reference atlas onto a new query dataset. Foundation models pretrained on large atlases like the Human Cell Atlas can perform label transfer in zero-shot or few-shot settings by comparing the learned representations of query cells to their reference embeddings, dramatically accelerating annotation workflows.

The computational reconstruction of transcription factor–target gene interactions from expression data. Foundation models learn these regulatory relationships implicitly during pretraining. Tools like SCENIC complement these models by identifying active regulons through co-expression analysis and cis-regulatory motif enrichment, providing mechanistic interpretability to the model's learned representations.