K-mer Tokenization

A k-mer is a contiguous subsequence of length k extracted from a DNA sequence. Tokenization proceeds by sliding a window of size k across the genome, typically with a stride of 1 for overlapping k-mers or a stride of k for non-overlapping segmentation.

• Overlapping (stride=1): Maximizes contextual coverage; a 1,000 bp sequence yields 994 tokens for k=7.
• Non-overlapping (stride=k): Reduces sequence length by a factor of k, lowering computational cost but potentially fragmenting motifs that span token boundaries.
• The resulting token is treated as a single atomic unit in the model's vocabulary.

The parameter k directly controls a critical trade-off between vocabulary cardinality and contextual information density.

• Small k (e.g., k=3): A vocabulary of only 4³ = 64 tokens. Each token captures minimal local context, forcing the model to rely heavily on the self-attention mechanism to build higher-order representations.
• Large k (e.g., k=7): A vocabulary of 4⁷ = 16,384 tokens. Each token inherently captures a richer local motif (e.g., a full transcription factor binding half-site), but the expanded vocabulary increases memory footprint and risks out-of-vocabulary tokens.
• Typical genomic models like DNABERT use k=6 (4,096 tokens) as a balanced starting point.

Standard DNA sequences contain ambiguous nucleotide codes defined by the IUPAC convention (e.g., 'N' for any base, 'R' for A or G). Tokenization strategies must explicitly handle these to avoid vocabulary explosion.

• Exclusion: K-mers containing non-ACGT characters are discarded, which can create artificial gaps in coverage.
• Canonicalization: Ambiguous bases are converted to a single canonical form (e.g., all 'N's become 'A'), preserving sequence contiguity at the cost of introducing artificial signal.
• Expanded Vocabulary: Ambiguity codes are treated as distinct characters, expanding the alphabet from 4 to 15 letters and dramatically increasing the theoretical vocabulary size.

DNA is double-stranded, meaning a sequence on the forward strand is equivalent to its reverse complement on the reverse strand. To enforce strand-agnostic learning, k-mer vocabularies are often collapsed.

• A canonical k-mer is defined as the lexicographically smaller of a k-mer and its reverse complement.
• This reduces the effective vocabulary size by nearly half (e.g., from 4,096 to ~2,080 for k=6).
• This preprocessing step prevents the model from learning strand-specific artifacts and improves generalization by treating 'ACGT' and its reverse complement 'ACGT' as the same token.

Fixed k-mer tokenization can be suboptimal because biological motifs have variable lengths. Byte-Pair Encoding (BPE) offers a data-driven alternative that learns a vocabulary of variable-length subwords.

• BPE starts with single nucleotides and iteratively merges the most frequent adjacent pairs.
• The resulting vocabulary naturally captures common motifs of varying lengths (e.g., 'TATA' box, 'CAAT' box) as single tokens, while rare sequences remain as shorter subword units.
• This approach, used in models like GenSLM, provides a more flexible tokenization that adapts to the statistical structure of the training genome.

Once the sequence is tokenized into k-mer IDs, each token is mapped to a dense vector via an embedding matrix of shape [vocab_size, hidden_dim].

• Random Initialization: Weights are learned from scratch during pre-training.
• Pre-trained Initialization: K-mer embeddings can be initialized from a pre-trained model like DNA2Vec, which uses a skip-gram approach to learn k-mer representations that capture co-occurrence statistics, giving the transformer a warm start.
• The embedding layer is the first and most memory-intensive layer in the model, as its size scales linearly with the vocabulary cardinality.

The fundamental process of segmenting raw nucleotide sequences into discrete units for model input. Strategies range from single-nucleotide tokenization (vocabulary size of 4: A, T, C, G) to k-mer tokenization and byte-pair encoding (BPE) . The choice directly impacts vocabulary size, sequence length, and the model's ability to capture local motifs.

• Single nucleotides: minimal vocabulary, longest sequences, no local context per token
• K-mers: balanced vocabulary, captures local patterns like dinucleotide repeats
• BPE: adaptive vocabulary, compresses common subsequences efficiently

A pioneering genomic language model that adapted the BERT architecture specifically for DNA sequences using k-mer tokenization. DNABERT demonstrated that treating DNA as a language with overlapping 3-mer to 6-mer tokens enables transformers to learn regulatory element syntax.

• Pre-trained on the human reference genome with masked language modeling
• Achieved state-of-the-art on promoter, splice site, and transcription factor binding site prediction
• Established the k-mer paradigm that influenced subsequent models like DNABERT-2 and the Nucleotide Transformer

The core computational operation that processes k-mer token embeddings by computing pairwise relevance scores between every position in a sequence. For a sequence tokenized into k-mers, self-attention allows the model to directly connect a distal enhancer token with a promoter token thousands of positions away.

• Computes query, key, and value projections for each token
• Attention weights reveal which k-mers the model considers functionally related
• Enables capture of long-range cis-regulatory interactions without recurrent bottlenecks

An adaptive tokenization algorithm originally from NLP that has been applied to genomic sequences as an alternative to fixed k-mer tokenization. BPE starts with single nucleotides and iteratively merges the most frequent adjacent pairs to build a vocabulary of variable-length subwords.

• Automatically discovers common motifs like 'CG' dinucleotides or 'TATA' boxes as single tokens
• Balances vocabulary size against sequence length based on data statistics
• Used in models like GenSLM and BioBERT for adaptive genomic representation

A suite of state-of-the-art genomic foundation models that uses 6-mer tokenization with overlapping windows to process DNA from multiple species. The fixed k-mer approach enables the model to learn robust, transferable embeddings that capture evolutionary conservation patterns.

• Pre-trained on diverse genomes including human, mouse, and zebrafish
• 6-mer vocabulary of 4,096 tokens balances granularity with computational efficiency
• Embeddings transfer effectively to variant effect prediction and chromatin accessibility tasks

The self-supervised pre-training objective used alongside k-mer tokenization where a random subset of k-mer tokens is masked and the model must predict them from surrounding context. For genomic sequences, this forces the model to learn regulatory grammar and sequence conservation patterns.

• Typically masks 15% of k-mer tokens during training
• Model learns that certain k-mer patterns are statistically interdependent
• Enables zero-shot prediction of the impact of nucleotide variants on sequence likelihood