Nucleotide Transformer

Pre-trained on reference genomes from multiple species including human, mouse, and other organisms, enabling the model to learn evolutionarily conserved sequence patterns. This cross-species exposure creates embeddings that capture universal genomic grammar—such as splice sites, promoter motifs, and regulatory elements—that transfer effectively to species with limited training data. The diversity of training sequences prevents overfitting to species-specific quirks and produces representations that generalize across the tree of life.

Employs the masked language modeling (MLM) objective adapted for nucleotide sequences. During pretraining, random 6-mers (k-mers of length 6) are masked from input DNA sequences, and the model learns to predict the original nucleotides from surrounding context. This forces the transformer to build rich internal representations of:

• Cis-regulatory grammar
• Coding vs. non-coding distinctions
• Splice junction architecture No labeled data is required, allowing the model to leverage the vast corpus of unannotated reference genomes.

DNA sequences are tokenized using non-overlapping 6-mers (hexamers), producing a vocabulary of 4,096 possible tokens (4^6). This tokenization strategy:

• Captures local sequence motifs like transcription factor binding sites
• Reduces sequence length by 6x compared to single-nucleotide tokenization, improving computational efficiency
• Preserves reading frame information critical for coding region analysis Each 6-mer is embedded into a continuous vector space, forming the input to the transformer encoder stack.

Fine-tuned Nucleotide Transformer models can predict the functional impact of genetic variants by comparing embeddings of reference and alternate alleles. The model's attention heads learn to focus on evolutionarily constrained regions, enabling zero-shot variant prioritization without task-specific training. Applications include:

• Missense variant pathogenicity scoring
• Splice-altering variant detection
• Regulatory variant impact assessment This capability leverages the model's deep understanding of sequence conservation learned during pretraining.

When fine-tuned on ENCODE and GTEx data, the Nucleotide Transformer predicts epigenomic tracks (chromatin accessibility, histone modifications) and gene expression levels directly from DNA sequence. The model's multi-head self-attention mechanism captures long-range interactions up to tens of kilobases, enabling it to model enhancer-promoter looping and other distal regulatory relationships. Performance rivals specialized architectures like Enformer and Basenji while requiring less task-specific architectural engineering.

The transformer architecture provides built-in interpretability through attention weight analysis. Researchers can extract:

• Attention maps showing which genomic regions influence predictions
• Nucleotide-level importance scores via gradient-based attribution methods like Integrated Gradients
• Motif discovery by clustering attention patterns that correspond to known transcription factor binding sites This transparency is critical for regulatory genomics applications where understanding the mechanistic basis of predictions is as important as accuracy.

Fine-tuned Nucleotide Transformer embeddings can predict RNA transcript abundance directly from promoter and enhancer sequences. By leveraging self-supervised representations of regulatory grammar learned during pre-training, these models forecast expression levels across diverse tissues and cell types with fewer training examples than task-specific architectures.

• Predicts TPM and FPKM values from flanking sequence
• Captures long-range enhancer-promoter interactions
• Outperforms models trained solely on expression data when fine-tuned on matched GTEx samples

Nucleotide Transformer embeddings encode functional constraints on genomic sequence, enabling accurate prediction of whether single nucleotide variants (SNVs) alter molecular phenotypes. By comparing wild-type and mutated sequence representations, models score the pathogenicity of non-coding variants implicated in disease.

• Prioritizes regulatory variants from GWAS catalogs
• Computes delta scores between reference and alternate alleles
• Identifies deleterious mutations in deep intronic and intergenic regions missed by coding-only tools

The model predicts epigenomic tracks including DNase-seq, ATAC-seq, and ChIP-seq signals for hundreds of transcription factors and histone modifications. Nucleotide Transformer's multi-species pre-training enables cross-species transfer of regulatory logic.

• Predicts chromatin accessibility peaks from sequence alone
• Identifies transcription factor binding motifs without position weight matrices
• Generalizes to unseen cell types through embedding similarity

Embeddings from Nucleotide Transformer distinguish active promoters from poised or silent regions and identify distal enhancer elements. The model's attention heads naturally attend to core promoter motifs like TATA boxes, INR elements, and CpG islands.

• Binary classification of genomic regulatory elements
• Tissue-specific enhancer activity prediction
• Zero-shot transfer to non-human species for regulatory annotation

Fine-tuned Nucleotide Transformer models accurately identify donor and acceptor splice junctions by recognizing the degenerate sequence motifs that define exon-intron boundaries. The model captures dependencies between the branch point, polypyrimidine tract, and terminal dinucleotides.

• Predicts canonical and non-canonical splice sites
• Detects cryptic splice sites activated by deep intronic variants
• Complements specialized tools like SpliceAI with broader genomic context

Because Nucleotide Transformer is pre-trained on diverse species including human, mouse, zebrafish, and plants, its embeddings transfer across the tree of life. This enables annotation of newly sequenced genomes without species-specific training data.

• Cross-species regulatory element discovery
• Conservation analysis through embedding similarity
• Functional annotation of non-model organism genomes using human-trained classifiers