Nucleotide Transformer

Unlike models trained solely on the human genome, the Nucleotide Transformer is pre-trained on a diverse corpus of DNA sequences spanning multiple species across the tree of life. This cross-species exposure enables the model to learn universal genomic grammar—fundamental patterns of sequence organization, codon usage, and regulatory syntax that are conserved across evolution. The resulting embeddings capture both species-specific and evolutionarily conserved features, making the model highly transferable to non-human genomic tasks.

• Trained on genomes from 850+ species in the most comprehensive variant
• Learns conserved regulatory motifs shared across vertebrates, plants, and microorganisms
• Enables cross-species transfer learning for organisms with limited labeled data

The core innovation of the Nucleotide Transformer is its ability to generate contextualized representations of individual nucleotides. Unlike static one-hot encodings or fixed k-mer frequencies, each nucleotide's embedding is dynamically influenced by the surrounding sequence context—potentially spanning tens of thousands of base pairs. This allows the model to distinguish between identical sequence motifs that have different functions depending on their genomic neighborhood, such as a TATA box in a promoter versus the same sequence in a non-functional region.

• Captures long-range dependencies up to 100+ kilobases
• Distinguishes functionally identical motifs by genomic context
• Embeddings encode epigenomic state and chromatin accessibility signals

A powerful emergent capability of the Nucleotide Transformer is zero-shot mutation scoring. By computing the difference in sequence likelihood between a reference allele and an alternative allele, the model can predict the functional impact of genetic variants without any task-specific fine-tuning. This approach has demonstrated competitive performance with supervised methods on clinically relevant benchmarks like ClinVar and gnomAD constraint metrics, making it a valuable tool for prioritizing variants of unknown significance in rare disease diagnostics.

• Scores single-nucleotide variants and small indels
• Correlates with evolutionary conservation and pathogenicity annotations
• Applicable to non-coding regulatory variants that traditional tools miss

The self-attention mechanism within the Nucleotide Transformer provides a built-in interpretability framework. By visualizing attention heatmaps, researchers can identify which specific nucleotides the model focuses on when making predictions. These attention patterns often correspond to known transcription factor binding sites, splice junctions, and other functional elements. This enables de novo motif discovery—the identification of novel regulatory sequences directly from the model's learned representations without prior biological knowledge.

• Attention heads specialize in different motif types (GC-rich, AT-rich, palindromic)
• Enables discovery of uncharacterized regulatory elements
• Provides mechanistic interpretability for regulatory predictions

To make adaptation accessible for research labs with limited compute, the Nucleotide Transformer is compatible with Parameter-Efficient Fine-Tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) and adapter modules. These techniques update only a tiny fraction of the model's parameters—often less than 1%—while freezing the pre-trained backbone. This enables rapid specialization for downstream tasks like cell-type-specific expression prediction or enhancer-gene linking without the prohibitive cost of full fine-tuning.

• LoRA adapters require training only ~0.1% of parameters
• Enables fine-tuning on single consumer GPUs
• Preserves general genomic knowledge while learning task-specific features

The Nucleotide Transformer models are released under permissive open-source licenses and are available through the Hugging Face Model Hub. Multiple model sizes are provided—ranging from 500M to 2.5B parameters—allowing researchers to balance performance against computational constraints. Pre-computed embeddings for common reference genomes are also distributed, enabling bioinformaticians to leverage the representations without running the full model. This open ecosystem has fostered a growing community of tools and benchmarks built around the model.

• Available on Hugging Face with standard Transformers API
• Multiple sizes: 500M, 1B, and 2.5B parameters
• Pre-computed embeddings for GRCh38 and other reference genomes

Leverage learned evolutionary constraints to score the functional impact of single-nucleotide polymorphisms (SNPs). The model distinguishes benign from pathogenic variants by evaluating how a mutation alters the sequence likelihood under the learned genomic grammar.

• Zero-shot capability: Predicts mutation effects without any supervised fine-tuning on labeled variant data.
• ClinVar benchmarking: Achieves competitive performance against specialized tools like CADD and PrimateAI.
• Mechanism: A significant drop in log-likelihood at a mutated position signals a disruptive, likely deleterious change.

Classify genomic regions into functional regulatory elements directly from sequence context. The model's multi-head attention captures the complex syntax of transcription factor binding motifs and their combinatorial grammar.

• Core promoter detection: Identifies TATA boxes, initiator elements, and downstream promoter elements.
• Distal enhancer mapping: Recognizes clusters of binding sites for pioneer factors and signal-dependent transcription factors.
• Cross-tissue generalization: Embeddings transfer across cell types, reducing the need for tissue-specific training data.

Predict which regions of the genome are in an open chromatin state and accessible for transcription factor binding. The model learns the sequence determinants of DNase I hypersensitivity and ATAC-seq peaks.

• Cell-type-specific inference: Fine-tune with a lightweight adapter to predict accessibility in diverse cell lines.
• Footprinting resolution: Attention weights resolve individual protected binding sites within broader accessible domains.
• Benchmark performance: Outperforms convolutional baselines on the Genomic Benchmarks human_enhancers_cohn and human_ocr_ensembl tasks.

Transfer regulatory annotations from well-characterized model organisms to non-model species with sparse data. The foundational genomic grammar learned during pre-training on multiple species enables robust cross-species generalization.

• Human to mouse: Fine-tune a human-heavy pre-trained model on limited mouse ChIP-seq data to identify conserved enhancers.
• Agricultural genomics: Apply models to crop genomes like maize and rice for trait-associated regulatory variant discovery.
• Evolutionary conservation signal: The model implicitly learns phylogenetic constraints, aligning functionally equivalent elements across divergent clades.

Predict exon-intron boundaries and alternative splicing events from raw genomic sequence. The model learns the canonical dinucleotide signals (GT-AG) and the broader splicing code, including branch points and polypyrimidine tracts.

• Donor/acceptor scoring: Assigns precise probability scores to candidate splice junctions.
• Aberrant splicing detection: Flags cryptic splice sites activated by deep intronic mutations.
• Isoform prediction: Contextual embeddings inform models that predict tissue-specific exon inclusion and skipping patterns.

Predict enhancer-promoter looping and topologically associating domain (TAD) boundaries from linear sequence. The transformer's long-range self-attention captures the sequence grammar of CTCF and cohesin binding that organizes chromatin architecture.

• Hi-C contact imputation: Predicts contact frequency maps from DNA sequence alone.
• Insulator element detection: Identifies CTCF binding sites with convergent motif orientation that demarcate TAD boundaries.
• Structural variant impact: Assesses how genomic rearrangements disrupt normal 3D folding and cause enhancer hijacking.