Epigenomic Foundation Models

These models are pre-trained on millions of unlabeled DNA sequences from hundreds of species and cell types using self-supervised objectives like masked language modeling. By predicting masked nucleotides from surrounding sequence context, the model learns fundamental regulatory syntax—promoters, enhancers, splice sites—without requiring expensive, manually curated labels. This phase ingests terabytes of raw sequencing data, building a rich internal representation of genomic grammar.

A single pre-trained foundation model can be fine-tuned to predict chromatin accessibility, histone modifications, DNA methylation, and gene expression across diverse tissues and organisms. This cross-task transfer learning is enabled by the model's learned latent representations of regulatory elements, which are conserved across biological contexts. A model trained primarily on human data can generalize to mouse or even plant regulatory prediction tasks with minimal additional training.

Unlike early convolutional models limited to a few kilobases, modern epigenomic foundation models employ transformer attention mechanisms or dilated convolutions to capture regulatory interactions across 100-200 kilobases. This long-range receptive field is critical for modeling enhancer-promoter looping, where a regulatory element can control a gene located far away on the linear chromosome. The Enformer model, for example, uses a 200 kb input window to predict gene expression by integrating distal regulatory signals.

A powerful emergent capability: foundation models can predict the functional impact of non-coding genetic variants without being explicitly trained on variant data. By performing in-silico mutagenesis—computationally introducing a single nucleotide change and measuring the difference in predicted epigenomic tracks—the model quantifies the regulatory consequence of mutations. This enables prioritization of disease-associated variants from genome-wide association studies (GWAS) that fall in non-coding regulatory regions.

Foundation models pre-trained on a broad compendium of cell types can be fine-tuned to make accurate predictions for rare or difficult-to-assay cell types with limited training data. This addresses a critical bottleneck in epigenomics: generating high-quality data for every cell type of interest is experimentally prohibitive. The model leverages shared regulatory logic learned from abundant cell types to bootstrap predictions for scarce ones, enabling chromatin profile imputation for unmeasured conditions.

The internal embeddings of epigenomic foundation models encode biologically meaningful features. By probing the model's latent space, researchers can extract learned representations of transcription factor binding motifs, splice junctions, and chromatin states. Techniques like integrated gradients and attention weight analysis reveal which input nucleotides drive specific predictions, transforming the model from a black-box predictor into a discovery tool for novel regulatory sequence motifs.

A training methodology where models learn universal regulatory grammar from unlabeled genomic sequences before fine-tuning. Common pretext tasks include:

• Masked sequence modeling: predicting masked nucleotides from surrounding context
• Next token prediction on tokenized genomic sequences
• Contrastive learning across species or cell types
• This approach enables models to leverage massive unlabeled datasets, reducing reliance on scarce, expensive epigenomic assays.

A neural network training strategy where a single model simultaneously predicts multiple epigenomic assays across different cell types. Benefits include:

• Shared biological representations that improve generalization to rare assays
• Reduced overfitting through implicit regularization across tasks
• Efficient parameter usage compared to training separate models
• Enables cross-cell-type transfer learning when some assays are missing for specific cell types

A computational perturbation technique that systematically introduces virtual mutations into a DNA sequence to quantify their predicted impact. Applications include:

• Saturation mutagenesis: testing all possible single-nucleotide variants at each position
• Prioritizing non-coding variants from GWAS studies
• Identifying causal regulatory motifs driving epigenomic predictions
• Generating mutation effect maps that visualize sequence determinants of regulatory activity

The process of adapting a model pre-trained on a large, general epigenomic corpus to a specific, data-scarce target task. Typical workflow:

• Pre-train on comprehensive reference epigenomes (ENCODE, Roadmap)
• Fine-tune on a rare cell type or disease state with limited assay data
• Freeze early layers and adapt only task-specific heads
• Leverages learned universal regulatory grammar to overcome data scarcity

The statistical assessment of a model's confidence in its epigenomic predictions. Distinguishes between:

• Epistemic uncertainty: model ignorance due to limited training data, reducible with more data
• Aleatoric uncertainty: inherent noise in the biological measurement, irreducible
• Techniques include Monte Carlo dropout, deep ensembles, and evidential regression
• Critical for clinical applications where high-confidence predictions are required for variant interpretation