DNA Tokenization

The most granular approach where each base (A, T, C, G) is treated as an individual token, resulting in a minimal vocabulary of just 4 tokens plus special characters. This strategy preserves single-nucleotide resolution and allows the model to learn all possible sequence motifs directly from raw data. However, it produces extremely long token sequences—a single human chromosome can generate over 100 million tokens—requiring models with very large context windows or sparse attention mechanisms. The Nucleotide Transformer and Enformer architectures use this approach, relying on self-attention to build higher-level k-mer representations internally.

Splits the genome into overlapping or non-overlapping subsequences of fixed length k, typically ranging from 3 to 6 nucleotides. A 6-mer vocabulary contains 4^6 = 4,096 possible tokens, dramatically reducing sequence length while encoding local sequence context directly into each token. DNABERT popularized this approach using 6-mers, demonstrating that k-mer tokenization enables the model to capture short regulatory motifs like transcription factor binding sites more efficiently. The trade-off is a larger embedding matrix and potential loss of resolution at k-mer boundaries.

Adapts the Byte-Pair Encoding (BPE) algorithm from natural language processing to learn an optimal subword vocabulary directly from genomic data. Starting with single nucleotides, BPE iteratively merges the most frequent adjacent token pairs, creating a vocabulary that balances frequency and length. This data-driven approach automatically discovers common biological motifs—such as CpG islands, restriction sites, and repetitive elements—as multi-nucleotide tokens. The resulting vocabulary is more compact than fixed k-mers while preserving the ability to represent rare or novel sequences through individual base tokens.

A critical design decision in k-mer tokenization. Overlapping k-mers slide a window one nucleotide at a time, ensuring every possible subsequence is represented and preserving translational invariance—the same motif at different positions receives identical tokenization. Non-overlapping k-mers segment the sequence into contiguous blocks, reducing the total token count by a factor of k but introducing boundary artifacts where biologically meaningful motifs may be split across tokens. Most genomic language models, including DNABERT, use overlapping tokenization to maintain positional consistency of regulatory elements.

Genomic tokenization requires specialized tokens beyond the nucleotide vocabulary. Classification tokens ([CLS]) aggregate sequence-level representations for downstream tasks like enhancer prediction. Mask tokens ([MASK]) enable self-supervised pre-training objectives. Critically, many models incorporate strand-specific tokens or reverse-complement augmentation to account for the double-stranded nature of DNA, where a regulatory motif on the forward strand has an equivalent representation on the reverse strand. Without this, the model must learn strand invariance from data alone, increasing training complexity.

The tokenization strategy directly determines how much genomic territory a model can observe within its fixed context window. A model with a 512-token context window using single nucleotides sees only 512 base pairs, insufficient for capturing distal enhancer-gene interactions. The same model using 6-mer tokenization sees 3,072 base pairs. Enformer addresses this by combining single-nucleotide tokenization with an exceptionally large receptive field through hierarchical attention, while HyenaDNA uses implicit long convolutions to process up to 1 million nucleotides at single-base resolution without tokenization.

The most prevalent DNA tokenization strategy that segments a genome into fixed-length subsequences of length k. Overlapping k-mers (e.g., sliding a window of 6 nucleotides one base at a time) capture rich contextual information but create a large, redundant vocabulary. Non-overlapping k-mers reduce sequence length but lose boundary information. The choice of k represents a critical trade-off: smaller values (k=3-4) yield compact vocabularies but short context, while larger values (k=6-8) capture motifs like restriction enzyme sites but risk out-of-vocabulary tokens on rare sequences.

An adaptive tokenization algorithm borrowed from natural language processing that iteratively merges the most frequent pairs of nucleotides or k-mers into new tokens. Unlike fixed k-mer tokenization, BPE learns a data-driven vocabulary where common biological motifs (e.g., TATA box, CpG islands) become single tokens, while rare variants remain as individual bases. This produces a variable-length token set that balances vocabulary size with semantic density, dramatically reducing sequence length for transformer processing while preserving biologically meaningful units.

A family of genomic foundation models that ingests tokenized DNA sequences to produce contextual nucleotide embeddings. Pre-trained on diverse genomes across multiple species using a standard k-mer tokenizer (k=6), these models learn representations that transfer robustly to downstream tasks. The tokenization scheme ensures that the same k-mer appearing in different genomic contexts—such as a coding exon versus an intergenic region—receives different embeddings based on surrounding sequence, capturing positional regulatory grammar.

A pioneering genomic language model that directly adapted the BERT architecture for DNA by replacing word tokenization with overlapping k-mer tokenization (k=3 to 6). The tokenizer treats each k-mer as a discrete vocabulary entry, and the model is pre-trained using masked language modeling on the human reference genome. DNABERT demonstrated that tokenization choice directly impacts downstream performance: k=6 captured dinucleotide frequency patterns and splice junction motifs better than single-nucleotide tokenization for tasks like promoter prediction.

The core architectural component that consumes tokenized DNA sequences and computes pairwise relevance scores between every position. After tokenization converts nucleotides into discrete vocabulary indices, an embedding layer maps each token to a dense vector. Self-attention then allows the model to weigh the importance of every other token when building a representation for a given position. This is how a model learns that a token 50,000 bases away at an enhancer is relevant to a token at a promoter, capturing long-range cis-regulatory interactions.

A standardized evaluation framework that exposes how tokenization choices impact model performance across diverse tasks. The benchmark suite includes tasks like human enhancer identification, chromatin accessibility prediction, and splice site detection. Experiments reveal that BPE tokenization often outperforms fixed k-mer tokenization on tasks requiring motif-level understanding, while single-nucleotide tokenization with a large transformer can match k-mer methods given sufficient compute. These benchmarks provide the empirical foundation for tokenizer selection in production genomic ML pipelines.