Single-Cell Foundation Model

These models are trained on datasets containing tens of millions of single-cell transcriptomes using self-supervised objectives that do not require manual labels. The model learns to predict masked gene expression values or reconstruct corrupted input data, forcing it to internalize the underlying grammar of gene regulation. For example, Geneformer was pretrained on approximately 30 million human single-cell transcriptomes from diverse tissues, while scGPT leverages a generative pretraining objective on over 33 million cells. This scale enables the model to encounter a vast diversity of cell types, states, and conditions, building a robust foundational understanding of cellular biology that generalizes across tissues and species.

Unlike static gene embeddings, foundation models generate dynamic, context-dependent representations where the vector for a gene like TP53 shifts based on the surrounding transcriptomic environment. This is achieved through the attention mechanism, which weighs the influence of every other gene when encoding a specific gene's expression level. The result is a high-dimensional embedding space where cells cluster by biological function rather than technical artifacts, and gene-gene relationships reflect regulatory logic rather than simple co-expression. These embeddings serve as universal features for downstream tasks without requiring task-specific architecture changes.

A defining capability of foundation models is the ability to perform zero-shot inference on entirely new cell types or conditions never seen during training. By fine-tuning with as few as dozens to hundreds of labeled examples, the model adapts to specialized tasks such as:

• Predicting disease-specific transcriptional perturbations
• Annotating rare cell populations in novel tissues
• Identifying drug response signatures in patient-derived samples This dramatically reduces the experimental and computational cost of building bespoke models for each new study, enabling rapid hypothesis generation in translational research.

A single pretrained model can be fine-tuned for a wide spectrum of biological tasks without architectural modification:

• Cell-type annotation: Classifying cells into known or novel types
• Gene regulatory network inference: Predicting which transcription factors regulate target genes
• Perturbation response prediction: Forecasting transcriptomic changes after CRISPR knockouts or drug treatments
• Batch integration: Harmonizing data from multiple experiments without explicit batch correction This versatility stems from the model's deep internalization of gene expression syntax during pretraining, making it a general-purpose engine for single-cell analysis.

Foundation models trained primarily on human or mouse data demonstrate remarkable cross-species generalization, successfully annotating cell types in zebrafish, drosophila, and other model organisms. This transfer works because the fundamental grammar of gene regulation—co-expression modules, regulatory cascades, and housekeeping programs—is evolutionarily conserved. Additionally, architectures are being extended to incorporate multi-omics modalities beyond transcriptomics, including chromatin accessibility (scATAC-seq) and surface protein abundance (CITE-seq), creating unified representations that capture multiple layers of cellular regulation simultaneously.

Beyond representation learning, these models possess generative capabilities that enable computational perturbation experiments. By conditionally generating gene expression profiles, researchers can simulate:

• Knockout phenotypes: Predict the transcriptomic outcome of deleting a specific gene
• Disease progression trajectories: Generate intermediate cell states along a pathological continuum
• Drug response profiles: Forecast how a cell's expression changes under chemical treatment This in silico experimentation accelerates hypothesis generation and reduces the need for costly wet-lab screening, particularly in early-stage drug discovery.

The foundational learning paradigm where a model is trained on massive, unlabeled single-cell data by solving a pretext task—such as masked gene prediction or next-token prediction—to learn intrinsic biological patterns. This eliminates the need for costly manual annotation. After pretraining, the model develops a universal understanding of gene co-expression, regulatory dynamics, and cellular states that generalizes across tissues and species.

A core architectural component that allows the model to dynamically weigh the importance of each gene in the context of all other genes within a cell's transcriptomic profile. Unlike static gene lists, multi-head self-attention captures context-dependent regulatory interactions—for example, recognizing that the significance of a transcription factor's expression depends entirely on the presence of its co-factors and target genes.

The process of converting raw gene expression values into discrete tokens that a transformer model can process, analogous to how words are tokenized in NLP. Strategies include:

• Value binning: Partitioning continuous expression into discrete bins
• Gene ranking: Ordering genes by expression magnitude
• Attention-based pooling: Learning continuous embeddings directly This step critically determines how the model perceives transcriptional semantics.

A capability where a pretrained foundation model can classify cell types it was never explicitly trained to recognize by leveraging its deep semantic understanding of transcriptional programs. By comparing the embedding of a query cell to reference embeddings of known cell types—or by probing the model with natural language descriptions of cell markers—the model generalizes to novel taxonomies without fine-tuning.

A critical preprocessing and architectural challenge. Technical variation from different laboratories, sequencing platforms, or protocols can confound biological signal. Foundation models address this through:

• Explicit batch tokens that condition the model on technical covariates
• Adversarial training to learn batch-invariant representations
• Domain adaptation techniques that align latent spaces across batches The goal is to isolate and remove non-biological variation while preserving true cellular heterogeneity.

A landmark single-cell foundation model developed by researchers at the Broad Institute, trained on approximately 30 million human single-cell transcriptomes using a masked gene prediction objective. Geneformer employs a context-aware, attention-based architecture that ranks genes by expression within each cell. It has demonstrated state-of-the-art performance on tasks including dosage-sensitive disease gene prediction, chromatin dynamics modeling, and cardiac lineage trajectory inference.