K-mer Tokenization

A sliding window of length k moves across the DNA sequence with a stride of 1, generating overlapping subsequences. Each unique k-mer becomes a token in the model's vocabulary. For example, with k=6, the sequence ATCGATCG yields tokens ATCGAT, TCGATC, and CGATCG. This ensures every nucleotide appears in k different tokens, providing redundant context that helps models learn robust representations of local sequence motifs.

The theoretical vocabulary size is 4^k for a standard nucleotide alphabet (A, T, C, G). For k=6, this yields 4,096 possible tokens. In practice, the vocabulary is built from k-mers observed in the training corpus, often augmented with special tokens:

• [CLS]: Aggregate sequence representation
• [SEP]: Boundary between sequences
• [MASK]: Token for masked language modeling
• [UNK]: Rare or unseen k-mers This structured vocabulary is the lookup table that maps biological sequence patterns to integer indices.

The stride determines the step size of the sliding window. A stride of 1 maximizes overlap, ensuring every nucleotide context is captured, but generates a token count nearly equal to sequence length. A stride of k produces non-overlapping tokens, reducing sequence length by a factor of k but losing boundary information. The choice trades off between computational efficiency and contextual resolution. Overlapping tokenization is standard for genomic language models to preserve regulatory element boundaries.

Real sequencing data contains IUPAC ambiguity codes (e.g., N for any base, R for purine). Tokenization strategies must handle these:

• Exclusion: Discard k-mers containing ambiguous bases
• Expansion: Replace ambiguity codes with all possible nucleotides, generating multiple tokens
• Special Token: Map any k-mer with an ambiguous base to a dedicated [UNK] or [AMBIG] token
• Probabilistic: Weight expanded tokens by base-calling quality scores The chosen strategy directly impacts downstream model robustness to sequencing errors.

DNA is double-stranded, so a sequence and its reverse complement encode equivalent biological information. Tokenization can enforce strand invariance by:

• Canonicalization: Always tokenize the lexicographically smaller of a k-mer and its reverse complement
• Dual Presentation: Tokenize both strands and let the model learn equivalence
• Augmentation: Randomly present either strand during training Canonicalization halves the effective vocabulary size and explicitly encodes the physical symmetry of DNA.

Each k-mer token ID is mapped to a dense vector via an embedding matrix of shape [vocab_size, hidden_dim]. This matrix is randomly initialized and learned during pretraining. The embedding space organizes k-mers by functional similarity—tokens representing binding motifs for the same transcription factor cluster together. For large k values (k>6), the vocabulary can exceed 65,000 tokens, requiring careful initialization and potential embedding factorization to manage memory.

A data compression algorithm adapted for tokenization that merges the most frequent pairs of bytes or characters to build a subword vocabulary. Unlike fixed-length k-mer tokenization, BPE dynamically constructs tokens of variable length based on statistical frequency in the training corpus.

• How it works: Starts with single nucleotides, iteratively merges the most frequent adjacent pairs
• Advantage: Captures common biological motifs (e.g., 'TATA', 'CGCG') as single tokens
• Trade-off: Vocabulary size is a hyperparameter; rare sequences remain fragmented
• Genomic application: Enables open-vocabulary handling of novel sequences without out-of-vocabulary tokens

The preprocessing component that segments a raw nucleotide string into a sequence of discrete integer tokens, forming the input layer for a DNA language model. The choice of tokenization strategy fundamentally shapes what patterns the model can learn.

• K-mer tokenization: Fixed-length sliding window, preserves local context
• BPE tokenization: Variable-length, frequency-adaptive subword units
• Character-level: Single nucleotide tokens, maximum flexibility but longer sequences
• Output: Integer IDs mapped through a learned embedding matrix to dense vectors
• Critical design choice: Determines sequence length, vocabulary size, and the granularity of learned representations

Dynamic nucleotide embeddings generated by a genomic language model where the vector for a given k-mer changes depending on its surrounding sequence context. This captures regulatory syntax—the same motif can have different meanings in different genomic neighborhoods.

• Static vs. contextual: A 'TATA' k-mer gets a different embedding in a promoter vs. an exon
• Mechanism: Self-attention layers aggregate information from all positions
• Biological significance: Enables the model to distinguish functional from non-functional instances of the same sequence motif
• Downstream use: These context-aware vectors serve as input features for variant effect prediction and regulatory element classification

Techniques used during self-supervised pretraining that deliberately introduce noise into input DNA sequences to force the model to learn robust, denoised representations. These strategies define the pretraining objective and what biological patterns the model internalizes.

• Masking: Random k-mers replaced with a [MASK] token (standard MLM)
• Deletion: Spans of sequence removed, model must reconstruct missing information
• Substitution: Nucleotides randomly swapped, model learns to identify unnatural patterns
• Span corruption: Contiguous blocks masked, encouraging long-range dependency learning
• Biological parallel: Analogous to the evolutionary processes of mutation and selection that shape real genomes

A data augmentation strategy that presents both strands of a DNA sequence during training, enforcing the model to learn strand-symmetric representations consistent with the double-helical nature of DNA. This is essential because regulatory elements can function on either strand.

• Mechanism: Each sequence is also presented as its reverse complement with the same label
• Biological basis: DNA is double-stranded; transcription factors bind regardless of strand orientation
• Implementation: Applied during k-mer tokenization—the reverse complement of each k-mer is mapped to a distinct but related token
• Benefit: Improves model generalization and reduces strand-specific artifacts in predictions