DNABERT

DNABERT segments raw nucleotide sequences into overlapping k-mers (typically k=3 to 6) using a sliding window with stride 1. This tokenization strategy captures sequence composition and local motif patterns that single-nucleotide tokenizers miss. Each k-mer is mapped to a unique integer ID in a fixed vocabulary, enabling the model to learn embeddings for short sequence motifs rather than individual bases. The overlap ensures that every nucleotide contributes to multiple tokens, providing redundant encoding that improves robustness to single-base mutations.

DNABERT adopts the BERT-base architecture with 12 transformer layers, 768 hidden dimensions, and 12 attention heads. Unlike unidirectional models, the bidirectional self-attention mechanism allows each k-mer token to attend to both upstream and downstream context simultaneously. This is critical for genomics because regulatory elements like promoters and enhancers are defined by flanking sequence context in both directions. The architecture captures long-range dependencies up to 512 tokens, corresponding to approximately 510 base pairs of input sequence.

DNABERT is pre-trained using a masked language modeling (MLM) objective adapted for DNA. During training, 15% of input k-mers are randomly masked, and the model must predict the original nucleotide sequence from the surrounding bidirectional context. This forces the model to learn the statistical grammar of the genome—including transcription factor binding motifs, splice sites, and coding potential—without requiring labeled data. The pre-training corpus is the human reference genome (GRCh38), providing a comprehensive foundation for downstream fine-tuning tasks.

The key output of DNABERT is a 512-dimensional embedding vector for each input k-mer that encodes its functional role within the specific sequence context. Unlike static embeddings (e.g., DNA2Vec), DNABERT produces dynamic representations where the same k-mer can have different embeddings depending on surrounding nucleotides. This context-sensitivity enables the model to distinguish between identical motifs that play different regulatory roles based on their genomic neighborhood—a critical capability for predicting cell-type-specific enhancer activity and transcription factor binding.

DNABERT supports transfer learning by attaching a task-specific classification head to the final hidden state of the [CLS] token. This pooled representation aggregates information from all k-mers into a fixed-length vector suitable for downstream tasks. The architecture has been fine-tuned for:

• Promoter prediction: Binary classification of whether a sequence contains a promoter region
• Transcription factor binding: Predicting binding sites for hundreds of TFs
• Splice site detection: Identifying exon-intron boundaries
• Chromatin state prediction: Classifying regulatory element types

DNABERT incorporates reverse complement awareness during pre-training by treating the forward and reverse complement strands as distinct but related sequences. This is essential because regulatory proteins can bind to either strand, and the model must learn strand-invariant features for some tasks while preserving strand-specific signals for others. The k-mer vocabulary includes both a sequence and its reverse complement as separate tokens, allowing the attention mechanism to learn the relationship between complementary motifs during pre-training.