Nucleotide Transformer

Models learn intrinsic genomic grammar from unlabeled DNA sequences via pretext tasks like masked language modeling (MLM). By predicting masked nucleotides from sequence context, the model builds a rich internal representation of regulatory syntax, splice sites, and coding potential without requiring curated labels.

Pre-trained on reference genomes spanning hundreds of species—from human and mouse to diverse clades across the tree of life. This cross-species exposure enables the model to capture evolutionarily conserved regulatory motifs and generalize to non-model organisms with limited annotation data.

Unlike static k-mer encodings, each nucleotide receives a dynamic vector representation informed by up to thousands of base pairs of surrounding context. These embeddings capture:

• Promoter-enhancer syntax
• Splice junction signals
• Open chromatin signatures
• Transcription factor binding motifs

Standard transformers are limited by quadratic attention complexity. Nucleotide Transformers employ sparse attention mechanisms or dilated attention windows to process sequences spanning 6,000 to 100,000 nucleotides, capturing distal regulatory interactions like enhancer-promoter looping that shorter context models miss.

Pre-trained embeddings serve as a frozen feature extractor or are fine-tuned end-to-end on downstream tasks:

• Chromatin accessibility prediction (ATAC-seq, DNase-seq)
• Histone modification calling (ChIP-seq)
• DNA methylation inference
• Variant effect prediction This dramatically reduces the labeled data required per task.

DNA is tokenized using overlapping k-mer vocabularies (typically k=6) rather than single nucleotides. This balances vocabulary size against sequence resolution. The byte-pair encoding (BPE) approach adapts token frequency to the genomic corpus, efficiently representing common repetitive elements and rare functional motifs.

Adapt transformer architectures for genomic sequences by treating nucleotides as tokens. These models use self-supervised learning to predict masked bases or next tokens, building internal representations of regulatory grammar, splice sites, and evolutionary constraints without explicit labels.

A training paradigm where models learn regulatory grammar from unlabeled DNA via pretext tasks:

• Masked Language Modeling: Predict masked nucleotides from context
• Next Token Prediction: Forecast subsequent bases autoregressively
• Contrastive Learning: Align embeddings of homologous sequences This pre-training enables strong performance on downstream tasks with limited labeled data.

A deep learning paradigm predicting genome-wide epigenomic tracks—chromatin accessibility, histone marks, transcription factor binding—directly from raw DNA. The Nucleotide Transformer embeddings serve as input features, enabling models to generalize regulatory predictions across unseen cell types and species.

A computational perturbation technique that systematically introduces virtual mutations into a DNA sequence to quantify their predicted impact on model output. By comparing wild-type and mutant predictions, researchers score variant pathogenicity and identify regulatory motifs critical for epigenomic function.

A model interpretability method attributing predictions to input nucleotides by accumulating gradients along a path from a neutral baseline to the actual sequence. This satisfies the completeness axiom, ensuring attribution scores sum to the prediction difference, and reveals which bases drive binding or accessibility predictions.

The process of adapting a model pre-trained on massive, diverse genomic corpora to a specific, data-scarce target task. The Nucleotide Transformer provides frozen or fine-tuned embeddings that capture universal sequence features, dramatically reducing the labeled data required for rare cell types or disease-specific epigenomic predictions.