Byte-Pair Encoding (BPE)

BPE builds a token vocabulary iteratively from the training data itself, starting with individual nucleotides (A, T, C, G, N) and repeatedly merging the most frequent adjacent token pair. This bottom-up process discovers biologically meaningful subword units—common dinucleotides like 'CG', trinucleotide motifs, and longer conserved elements—without requiring prior biological knowledge. The algorithm terminates when a predefined vocabulary size is reached, typically between 4,096 and 32,768 tokens for genomic models.

• Starting vocabulary: Single nucleotides + unknown token
• Merge operation: Combine most frequent adjacent pair
• Stopping criterion: Target vocabulary size reached
• Result: Hierarchical vocabulary of subword units

Unlike fixed k-mer tokenization, BPE provides complete coverage of any input sequence by recursively segmenting unseen character combinations into known subword units. A novel sequence variant is decomposed into its constituent merges until reaching the base vocabulary. This guarantees that no nucleotide sequence is ever out-of-vocabulary (OOV), a critical property for variant-rich genomic regions and cross-species applications where fixed k-mer vocabularies would fail on novel subsequences.

• OOV rate: Effectively zero
• Fallback mechanism: Recursive decomposition to single nucleotides
• Advantage over k-mers: Handles novel mutations and structural variants
• Cross-species robustness: Adapts to unseen genomic contexts

BPE automatically allocates shorter token sequences to frequent genomic patterns and longer token sequences to rare ones, achieving an implicit compression that mirrors natural sequence statistics. Highly conserved regulatory motifs like the TATA box or GC-rich promoter elements receive dedicated tokens, while rare or species-specific sequences are represented as combinations of more primitive units. This adaptive compression reduces sequence length by 2-4x compared to character-level encoding, directly lowering the quadratic attention cost in Transformer architectures.

• Compression ratio: 2-4x over character-level
• Frequent motifs: Assigned single-token representations
• Rare variants: Encoded as multi-token sequences
• Attention cost reduction: Proportional to compression ratio

The merge operations learned by BPE often correspond to biologically interpretable sequence features without explicit supervision. Early merges capture dinucleotide biases (CpG islands), intermediate merges recover known transcription factor binding motifs, and deeper merges can identify conserved regulatory modules and repetitive elements. This emergent property makes BPE not just a tokenization strategy but an unsupervised motif discovery tool, where the learned vocabulary itself provides insights into the sequence grammar of the training genome.

• Early merges: Dinucleotide frequencies and CpG patterns
• Mid-level merges: Transcription factor binding motifs
• Deep merges: Conserved regulatory modules and repeats
• Interpretability: Vocabulary inspection reveals biological grammar

BPE tokenization is fully deterministic—the same trained vocabulary and merge order always produces identical token sequences for a given input. This property is essential for reproducible genomic analyses and model inference pipelines. Furthermore, the encoding is losslessly reversible: the original nucleotide sequence can be exactly reconstructed from its token sequence by concatenating the subword units. This bidirectional mapping ensures no information is discarded during preprocessing, unlike hashing-based or fixed-window approaches that may lose positional resolution.

• Determinism: Identical input always yields identical tokens
• Reversibility: Full nucleotide sequence recovery from tokens
• Reproducibility: Critical for clinical and regulatory applications
• No information loss: Lossless compression guarantee

The target vocabulary size serves as a direct tunable hyperparameter controlling the trade-off between sequence compression and model complexity. A smaller vocabulary (1K-4K tokens) produces longer token sequences but a compact embedding matrix, reducing memory footprint for edge deployment. A larger vocabulary (16K-32K tokens) achieves greater compression and captures longer motifs as single tokens but increases the embedding layer parameters. This explicit control allows practitioners to optimize for specific hardware constraints and sequence length requirements.

• Small vocabulary (1K-4K): Compact model, longer sequences, edge-friendly
• Large vocabulary (16K-32K): Better compression, richer motifs, GPU-optimized
• Embedding parameters: Vocabulary size × embedding dimension
• Tunable for deployment: Balance memory vs. sequence efficiency

A foundational method that segments a nucleotide sequence into overlapping subsequences of fixed length k. For example, the sequence ATCG with k=3 yields tokens ATC, TCG. This converts raw DNA into discrete units that serve as the initial input vocabulary for models like DNABERT. The choice of k represents a critical trade-off: larger k values capture longer motifs but explode vocabulary size, while smaller k values risk losing sequence specificity.

The preprocessing component that converts a raw nucleotide string into a sequence of discrete integer tokens. Tokenizers bridge the gap between biological sequence data and neural network inputs. Common strategies include:

• K-merization: Fixed-length, overlapping windows
• Byte-Pair Encoding (BPE): Data-driven subword merging
• SentencePiece: Language-agnostic subword modeling The tokenizer's vocabulary size and merge rules directly impact a model's ability to handle open-vocabulary sequences and rare genomic elements.

Dynamic nucleotide embeddings generated by a genomic language model where the vector for a given token changes depending on its surrounding sequence context. Unlike static embeddings where ATC always maps to the same vector, contextualized representations capture regulatory syntax—the same k-mer in a promoter versus an enhancer will have different embeddings. This property is fundamental to how models like Enformer and DNABERT understand genomic grammar.

A data augmentation strategy that presents both strands of a DNA sequence during training. Since DNA is double-stranded and a regulatory element on one strand implies function on the complementary strand, this technique enforces strand-symmetric representations. During tokenization, sequences are randomly replaced with their reverse complement, ensuring the model learns that ATCG and its complement CGAT carry equivalent biological meaning.

Techniques used during self-supervised pretraining that deliberately introduce noise into input DNA sequences to force the model to learn robust, denoised representations. Common strategies include:

• Masking: Replacing tokens with a [MASK] token (used in MLM)
• Deletion: Removing random nucleotides
• Substitution: Swapping nucleotides with random alternatives These strategies are central to Masked Language Modeling and Genomic Masked Autoencoders, teaching models to infer missing or corrupted genomic information from surrounding context.

A metric derived from a language model's cross-entropy loss that quantifies how surprised the model is by a given sequence. In genomics, low perplexity indicates the sequence conforms to learned patterns of natural DNA, while high perplexity flags unusual or non-functional regions. Perplexity is used to:

• Measure evolutionary constraint
• Identify functional elements
• Score variant pathogenicity It provides a zero-shot signal for sequence naturalness without task-specific training.