Self-Supervised Pretraining

The core pretext task adapted from natural language processing. A percentage of nucleotides in an input sequence—typically 15%—are randomly masked, and the model is trained to predict the original bases from the surrounding genomic context. This forces the model to learn regulatory grammar, motif dependencies, and evolutionary constraints without any labeled data. Models like DNABERT and the Nucleotide Transformer use this strategy to build contextualized representations of k-mers.

An autoregressive pretraining strategy where the model learns to predict the next nucleotide or k-mer in a sequence given all preceding elements. This objective excels at capturing directional dependencies and causal relationships along the genome, such as splice donor-acceptor site ordering and the sequential logic of cis-regulatory modules. It is particularly effective for generative tasks like designing synthetic enhancers or promoters.

A pretraining paradigm that learns representations by pulling together augmented views of the same genomic region while pushing apart representations of different loci. Augmentations can include reverse complementing, random k-mer substitution, or simulated sequencing noise. This approach builds embeddings that are invariant to strand orientation and robust to natural genetic variation, making it ideal for variant effect prediction and cross-species alignment.

Training a single foundation model on aligned genomes from multiple species—such as human, mouse, and zebrafish—injects a powerful evolutionary inductive bias. The model learns to distinguish conserved functional elements from neutrally evolving background sequence. This cross-species signal acts as a natural regularizer, dramatically improving performance on downstream tasks like enhancer prediction and pathogenicity scoring for rare variants where human-labeled data is scarce.

The primary output of pretraining: dense vector representations where each nucleotide or k-mer is encoded not in isolation, but in the full context of its surrounding sequence. Unlike static one-hot encodings, these embeddings capture position-dependent semantics—an adenine in a TATA box promoter has a different vector than an adenine in an exonic splice enhancer. These embeddings serve as the transferable foundation for fine-tuning on expression prediction tasks.

The bridge from pretraining to gene expression prediction. A pretrained genomic model is adapted by adding a regression head and training on paired DNA sequence and RNA-seq data from resources like GTEx. The model learns to map the learned sequence representations to transcript abundance. This two-stage pipeline—pretrain on the whole genome, fine-tune on tissue-specific expression—consistently outperforms models trained solely on labeled expression data.

The core pretext task adapted from NLP where a percentage of input nucleotides are randomly masked, and the model learns to predict the original bases from bidirectional context. In genomics, this forces the model to learn regulatory syntax, splice motifs, and evolutionary constraints without any labeled data. Variants include masking contiguous k-mers rather than individual bases to prevent the model from exploiting local nucleotide composition biases.

A bidirectional encoder architecture that applies the BERT framework directly to the human genome by treating non-overlapping k-mers as tokens. DNABERT learns contextualized nucleotide embeddings by solving a masked language modeling task on sequences up to 512 tokens long. The pre-trained model produces transferable features that achieve state-of-the-art performance on downstream tasks including promoter prediction, splice site detection, and transcription factor binding site identification after minimal fine-tuning.

A collection of foundation models pre-trained on diverse DNA sequences from multiple species, including the human reference genome and 3,202 other genomes. By training on this broad phylogenetic spectrum, the model captures both conserved regulatory elements and species-specific genomic features. The largest variant uses 2.5 billion parameters and processes sequences up to 6,144 tokens, producing embeddings that transfer effectively to variant effect prediction, chromatin profile forecasting, and gene expression estimation.

A self-supervised strategy that learns representations by pulling augmented views of the same genomic region together in embedding space while pushing apart views from different regions. Applied to DNA, augmentations include random cropping, nucleotide substitution, and reverse complementation. This approach excels at learning position-invariant regulatory features and has been shown to capture enhancer-promoter interactions without explicit chromatin contact data.

The two-stage workflow where a model is first pre-trained on massive unlabeled genomic corpora using self-supervision, then fine-tuned on smaller labeled datasets for specific tasks like expression prediction. The pre-training phase typically consumes weeks of GPU time on thousands of genomes, while fine-tuning requires only hours on task-specific data. This paradigm enables few-shot generalization to rare cell types and tissues where labeled training data is scarce.

A mechanism that injects sequential order information into transformer models so they can distinguish nucleotide positions along a chromosome. Unlike text, genomic sequences require encoding of absolute chromosomal coordinates and strand orientation. Advanced approaches use rotary position embeddings (RoPE) that naturally capture the relative distance between regulatory elements, enabling models to learn long-range enhancer-promoter loops spanning hundreds of kilobases.