Genomic Tokenizer

Segments a nucleotide sequence into overlapping subsequences of fixed length k using a sliding window. A sequence of length L produces L - k + 1 tokens.

• k=6 is common for models like DNABERT, balancing vocabulary size (4^6 = 4096 tokens) with motif capture
• Overlap ensures every nucleotide contributes to multiple tokens, providing redundancy
• Vocabulary is deterministic and complete, requiring no training phase
• Limitation: Cannot represent motifs longer than k without splitting them across tokens

An adaptive tokenization algorithm that iteratively merges the most frequent adjacent token pairs to build a subword vocabulary from a genomic training corpus.

• Data-driven: Vocabulary reflects true nucleotide co-occurrence statistics
• Variable-length tokens: Common motifs like TATA boxes become single tokens; rare sequences remain as individual nucleotides
• Open vocabulary: Handles novel sequences by falling back to single-nucleotide tokens
• Training required: Must learn merges on a representative genomic corpus before tokenization

Reserved vocabulary indices that provide structural metadata to the model beyond raw nucleotide content.

• [CLS]: Aggregated sequence representation for classification tasks
• [SEP]: Separates distinct genomic segments, such as paired enhancer-promoter sequences
• [MASK]: Replaces tokens during masked language modeling pretraining
• [PAD]: Ensures uniform sequence length during batch processing
• Strand tokens: Some architectures prepend '+' or '-' tokens to encode strandedness

DNA is double-stranded, and a tokenizer must account for the equivalence between a sequence and its reverse complement. Strategies include:

• Reverse complement augmentation: Presenting both strands during training to enforce strand-agnostic representations
• Canonical k-mer representation: Storing each k-mer as the lexicographically smaller of itself and its reverse complement, halving the effective vocabulary
• Strand-specific tokenization: Preserving strand information when directionality matters, such as in transcription orientation tasks

Each integer token ID maps to a dense, learnable vector through an embedding matrix of size V × d, where V is vocabulary size and d is model dimension.

• Random initialization: Embeddings are learned from scratch during pretraining
• Weight tying: The input embedding matrix is often shared with the output projection layer to reduce parameters
• Embedding dimension: Typically 512–1024 for genomic models, balancing expressiveness with memory constraints
• Position encoding: Added to token embeddings to inject sequential order information

The choice of tokenization strategy directly determines the minimum sequence context the model can perceive and the computational cost of processing long sequences.

• K-mer tokenization: A sequence of length L becomes ~L tokens, preserving positional granularity but increasing sequence length
• BPE tokenization: Compresses frequent motifs, reducing effective sequence length and enabling longer genomic context within fixed attention windows
• Trade-off: Larger tokens reduce compute but may blur fine-grained mutation signals; smaller tokens preserve resolution at higher cost

Segments a nucleotide sequence into overlapping subsequences of fixed length k, converting raw DNA into discrete integer tokens. A sequence ATCGATCG with k=3 produces tokens ATC, TCG, CGA, GAT, ATC, TCG.

• k=6 is common for DNA language models like DNABERT
• Smaller k captures local motifs; larger k captures longer-range patterns
• Vocabulary size grows exponentially: 4^k possible tokens
• Overlapping windows preserve local context at token boundaries

A data compression algorithm adapted for genomic tokenization that iteratively merges the most frequent pairs of tokens to build a subword vocabulary. Unlike fixed k-merization, BPE learns variable-length tokens from the data distribution.

• Discovers common biological motifs as single tokens (e.g., TATA box)
• Handles open-vocabulary sequences without out-of-vocabulary issues
• Vocabulary size is a tunable hyperparameter, not fixed by k
• Balances token granularity with sequence compression ratio

A preprocessing strategy that presents both strands of a DNA sequence during training, enforcing the model to learn strand-symmetric representations. Since DNA is double-helical, a regulatory element functions identically on either strand.

• Doubles effective training data without additional sequencing
• Prevents model from learning spurious strand-specific artifacts
• Critical for tasks like transcription factor binding prediction
• Implemented by tokenizing both the forward sequence and its reverse complement

Techniques used during self-supervised pretraining that deliberately introduce noise into input DNA sequences to force the model to learn robust, denoised representations.

• Masking: Replace tokens with a special [MASK] token (used in MLM)
• Deletion: Remove random tokens, requiring the model to infer missing context
• Substitution: Replace nucleotides with random alternatives
• Span corruption: Mask contiguous blocks to learn long-range dependencies
• These strategies prevent the model from memorizing and force learning of regulatory grammar

Dynamic nucleotide embeddings generated by a genomic language model where the vector for a given k-mer changes depending on its surrounding sequence context. Unlike static embeddings, these capture regulatory syntax.

• The token CACGTG (E-box motif) receives different embeddings in a promoter vs. an intron
• Enables the model to distinguish functional vs. non-functional instances of the same sequence
• Generated by the self-attention mechanism aggregating flanking context
• Foundation for downstream tasks like variant effect prediction

The initial phase of training a DNA language model on massive, unlabeled genomic corpora using self-supervised objectives to learn universal sequence representations before fine-tuning on specific downstream tasks.

• Uses objectives like Masked Language Modeling (MLM) or autoregressive prediction
• Training data spans multiple species and genomic regions
• Learns fundamental regulatory grammar: promoters, enhancers, splice sites
• Pretrained weights are then fine-tuned with labeled data for tasks like variant effect prediction
• Dramatically reduces the need for expensive labeled genomic datasets