Byte-Pair Encoding (BPE)

BPE initializes a vocabulary with the four nucleotide bases (A, C, G, T) and iteratively identifies the most frequently co-occurring adjacent pair of tokens in the training corpus. This pair is merged into a new subword unit and added to the vocabulary. The process repeats for a predefined number of merge operations, building a hierarchy of common genomic motifs. Example: If 'CG' is the most frequent pair, it becomes a single token. Subsequently, 'A' followed by 'CG' might merge into 'ACG'. This data-driven approach captures biological reality—CpG dinucleotides and longer motifs like TATA boxes emerge as distinct tokens without manual specification.

The primary hyperparameter in BPE is the total vocabulary size, which directly controls the trade-off between sequence resolution and context window efficiency. A small vocabulary (e.g., 4k tokens) produces short subword units, requiring more tokens to represent a gene but capturing fine-grained mutations. A large vocabulary (e.g., 32k tokens) encodes longer motifs as single tokens, compressing the sequence length and allowing transformer models to attend to wider genomic contexts within fixed attention windows. Key insight: For a 200k-base-pair input, a 4k vocabulary might produce 50k tokens, while a 32k vocabulary might produce 25k tokens, effectively doubling the receptive field.

Unlike fixed k-mer encoding, BPE is an open-vocabulary system. It can tokenize any sequence, including those containing novel mutations or sequencing errors, by falling back to individual nucleotide tokens. If a rare variant creates an unseen sequence of tokens, the BPE tokenizer recursively splits it into known subword units or, in the worst case, individual bases. This prevents out-of-vocabulary errors that plague fixed-vocabulary approaches. Practical benefit: A model trained on a reference genome can still process a patient sample with a private structural variant without crashing, maintaining robustness in clinical genomics pipelines.

The BPE merge order implicitly encodes a statistical prior about biological sequence importance. Frequent motifs—such as CpG islands, Alu repeats, and conserved promoter elements—are merged early and receive dedicated tokens. Rare or random sequences remain decomposed into smaller units. This mirrors the biological reality that repetitive and conserved elements often carry functional significance. Contrast with random initialization: Unlike learned token embeddings that start from scratch, BPE vocabulary provides a structured initialization that reflects the underlying k-mer frequency distribution of the genome, potentially accelerating model convergence during pre-training.

Standard BPE is strand-specific, but genomic applications often require strand invariance due to the double-helix structure. A common adaptation is to apply BPE to canonical k-mers—the lexicographically smaller of a k-mer and its reverse complement—before the merge process. Alternatively, the training corpus can be augmented with reverse complement sequences, forcing the algorithm to merge complementary pairs symmetrically. Implementation detail: This ensures that a regulatory element on the forward strand and its reverse complement on the opposite strand map to identical subword tokens, preserving the biological equivalence of double-stranded DNA.

BPE offers distinct advantages over fixed k-mer tokenization for genomic language models. Adaptive granularity: BPE assigns short tokens to variable regions and long tokens to conserved motifs, while k-mer uses uniform length. Vocabulary efficiency: A 32k BPE vocabulary can represent motifs of varying lengths, whereas a 32k k-mer vocabulary is limited to k=8 (4^8 = 65k, but practical vocabularies are smaller). Context compression: BPE tokens represent variable-length genomic units, compressing repetitive regions more aggressively than uniform k-mers, which waste token budget on repetitive sequence. Trade-off: BPE tokenization is deterministic after training, while k-mer is parameter-free and requires no corpus.

A foundational sequence vectorization technique that decomposes a nucleotide string into all possible substrings of length k, then maps each k-mer to a unique numerical identifier or frequency vector. Unlike BPE's data-driven vocabulary, k-mer encoding uses a fixed, exhaustive vocabulary of size 4^k. For k=6, this produces 4,096 possible tokens. The method is computationally simple but suffers from vocabulary explosion at larger k values and treats all k-mers as equally independent, ignoring the compositional subword structure that BPE explicitly captures.

The tokenization strategy used by the DNABERT genomic language model, which segments DNA sequences into overlapping k-mers rather than BPE's merged subword units. DNABERT uses a sliding window of k=6 to generate tokens, then applies Masked Language Modeling (MLM) to learn context-aware embeddings. The key distinction from BPE is that DNABERT's vocabulary is pre-defined and static, while BPE learns the vocabulary from data, adaptively merging frequent adjacent pairs to balance sequence granularity with context length.

A biologically-motivated tokenization strategy that segments mRNA transcripts or coding DNA sequences into non-overlapping triplets of nucleotides, directly aligning the vocabulary with the fundamental functional units of the genetic code. Each codon maps to one of 64 possible tokens. Unlike BPE, which is purely statistical, codon tokenization imposes a hard biological prior—the triplet reading frame—that preserves translational meaning. This approach is ideal for coding sequence analysis but fails to capture regulatory motifs that span non-coding regions with irregular boundaries.

A standardized representation that selects the lexicographically smaller of a k-mer and its reverse complement to collapse the strand-specific sequence space into a single, unambiguous feature. This preprocessing step explicitly accounts for the double-stranded nature of DNA, ensuring that a sequence and its Watson-Crick complement map to identical embedding vectors. BPE vocabularies can be similarly strand-normalized by applying canonicalization before or after the merge operations, reducing vocabulary size and enforcing a biologically meaningful invariance.

An architecture that learns genomic representations at multiple resolutions simultaneously—nucleotide, subword, codon, and gene level—by aggregating features from different convolutional kernel sizes or transformer layers. BPE naturally produces a multi-scale vocabulary where frequent motifs become single tokens while rare sequences remain fragmented. This hierarchical tokenization mirrors the biological hierarchy of DNA: individual bases combine into motifs, motifs into regulatory elements, and elements into functional domains.

A mechanism that injects information about the absolute or relative position of each token into the input embedding, enabling permutation-invariant transformer architectures to process sequential genomic data. After BPE tokenization produces a variable-length sequence of subword units, positional encoding ensures the model knows where each token appears. Common approaches include sinusoidal encodings, learned position embeddings, and Rotary Position Embedding (RoPE), which encodes relative position directly into the attention calculation for better length extrapolation on long genomic loci.