Nucleotide Transformer (NT)

NT models are pre-trained on a diverse corpus spanning 3,202 genomes across 850 species, including model organisms, plants, fungi, and bacteria. This multi-species training strategy forces the model to learn universal regulatory syntax rather than memorizing species-specific motifs. The resulting embeddings transfer effectively to downstream tasks in organisms never seen during training, enabling cross-species zero-shot prediction of promoters, enhancers, and splice sites without requiring species-specific fine-tuning.

Unlike genomic models that use convolutional layers or sparse attention, NT employs a vanilla BERT-style encoder with full bidirectional self-attention. Key architectural choices include:

• 6-mer tokenization with overlapping windows, producing a vocabulary of 4,096 tokens
• Rotary Position Embeddings (RoPE) for length extrapolation beyond the 1,000-token training context
• Multi-head self-attention that captures long-range regulatory interactions across the input sequence This standard design enables seamless integration with existing transformer tooling and transfer learning pipelines.

NT uses a masked language modeling (MLM) objective adapted for genomics. During pre-training, 15% of 6-mer tokens are randomly masked, and the model learns to predict the original nucleotide sequence from the surrounding bidirectional context. This self-supervised task teaches the model:

• Regulatory grammar: promoters, enhancers, and insulator syntax
• Coding potential: distinguishing exons from introns
• Evolutionary conservation: identifying functionally constrained elements The MLM objective requires no labeled data, enabling training on raw, unannotated genomes.

NT embeddings capture genetic variation at multiple scales without explicit variant encoding. The model's attention heads learn to attend to single nucleotide polymorphisms (SNPs), insertions/deletions (indels), and structural variants by detecting disruptions in the local sequence context. This enables:

• Variant effect prediction: quantifying the functional impact of non-coding mutations
• Allele-specific embedding: generating distinct representations for reference and alternate alleles
• Regulatory motif disruption scoring: identifying variants that break transcription factor binding sites Fine-tuning on variant effect datasets like DeepSEA or ClinVar achieves state-of-the-art performance.

NT provides built-in interpretability through its self-attention mechanism. The attention weights can be extracted and visualized to identify which genomic regions the model focuses on for a given prediction. This enables:

• Attention saliency maps: highlighting regulatory elements like promoters and enhancers
• Motif discovery: identifying recurrent sequence patterns that drive high attention scores
• Long-range interaction detection: revealing enhancer-promoter loops and chromatin contacts These interpretability features are critical for regulatory compliance and biological validation of model predictions.

NT models support Low-Rank Adaptation (LoRA) for adapting to new genomic assays and species without full fine-tuning. LoRA freezes the pre-trained weights and injects trainable low-rank decomposition matrices into the attention layers, reducing the number of trainable parameters by over 99%. Benefits include:

• Memory efficiency: fine-tune a 2.5B parameter model on a single GPU
• Task switching: maintain separate LoRA adapters for different downstream tasks
• Catastrophic forgetting prevention: preserve general genomic knowledge while specializing This approach is essential for labs with limited computational resources.

The self-supervised pre-training objective used by the Nucleotide Transformer. During training, a random subset of input tokens (nucleotide k-mers) is masked, and the model learns to predict the original nucleotide identity from the surrounding bidirectional context. This forces the model to learn regulatory syntax and evolutionary constraints without requiring labeled data. The NT models were trained on a diverse corpus spanning the tree of life, enabling species-agnostic embeddings.

A foundational genomic language model that directly inspired the Nucleotide Transformer architecture. DNABERT adapts the BERT encoder by tokenizing sequences into overlapping k-mers and pre-training via MLM on the human reference genome. Key differences from NT include:

• Single-species training corpus (human only)
• Smaller model capacity
• Limited cross-species transfer capability NT generalizes this approach to thousands of genomes across diverse clades.

A core capability enabled by the Nucleotide Transformer's training on diverse genomes. A model pre-trained on species with rich functional annotations (e.g., human, mouse) can be fine-tuned on a target species with limited data. The conserved regulatory grammar learned during pre-training transfers across evolutionary distances. This is critical for:

• Non-model organisms with sparse annotations
• Agricultural genomics
• Pathogen surveillance
• Conservation biology

A mechanism that injects sequence order information into the input embeddings. Since transformer self-attention is permutation-invariant, positional encoding is essential for modeling the sequential nature of DNA. The Nucleotide Transformer uses learned positional embeddings that allow the model to:

• Distinguish upstream from downstream regulatory elements
• Capture distance-dependent interactions
• Generalize to variable-length input sequences during inference

A special aggregation token prepended to the input sequence in the Nucleotide Transformer's BERT-style architecture. The final hidden state of this token serves as a fixed-length, pooled representation of the entire input sequence. This embedding vector is used for downstream tasks such as:

• Promoter strength prediction
• Splice site classification
• Chromatin state annotation
• Variant effect scoring

An architectural concept that captures genomic features at multiple resolutions simultaneously. The Nucleotide Transformer's deep transformer layers naturally learn a hierarchical representation:

• Lower layers: Local nucleotide patterns and motifs
• Middle layers: Regulatory grammar and syntax
• Upper layers: Long-range interactions and global context This multi-scale property enables a single model to serve tasks from variant calling to gene expression prediction.