Genomic Pretraining

The core learning signal is derived from the sequence itself, eliminating the need for expensive labeled data. Masked Language Modeling (MLM) corrupts input tokens and trains the model to reconstruct the original nucleotides from bidirectional context, capturing regulatory grammar. Autoregressive modeling predicts the next token unidirectionally, enabling sequence likelihood estimation. Sequence corruption strategies—including deletion, substitution, and span masking—force the model to learn robust, denoised representations of genomic syntax.

Raw nucleotide strings must be segmented into discrete tokens before model ingestion. K-mer tokenization segments sequences into overlapping fixed-length subsequences, converting DNA into a vocabulary of 4^k possible tokens. Byte-Pair Encoding (BPE) builds a subword vocabulary by merging frequent token pairs, handling open-vocabulary sequences efficiently. The genomic tokenizer forms the critical input layer, and strategies like reverse complement augmentation enforce strand-symmetric representations consistent with the double helix.

Biological function depends on long-range dependencies—interactions between genomic elements separated by vast linear distances, such as enhancers and their target promoters. Standard self-attention scales quadratically with sequence length, creating a computational bottleneck. Solutions include the Hyena Operator, which uses long convolutions and gating for subquadratic scaling, and the Mamba State Space Model, which offers linear-time sequence mixing with an input-dependent selection mechanism. FlashAttention accelerates exact attention by minimizing memory reads on GPU hardware.

Transformers are permutation-invariant and require explicit position information. Rotary Position Embedding (RoPE) encodes absolute position into the attention computation via rotation matrices, enabling models to extrapolate to sequence lengths unseen during training—a critical property for genomes of varying sizes. This allows a model pretrained on short contigs to generalize to full chromosomes without architectural modification.

Pretraining quality is measured intrinsically. Perplexity scoring quantifies how surprised the model is by held-out sequences; lower perplexity indicates better learning of genomic regularities. Sequence log-likelihood assigned by autoregressive models measures conformity to natural DNA patterns. For functional validation, zero-shot variant effect prediction uses the log-likelihood ratio between reference and alternate alleles to predict pathogenicity without any supervised fine-tuning on labeled variant data.

Cross-species transfer learning leverages evolutionary conservation by applying representations learned from one organism to another, improving performance on species with limited training data. The Evolutionary Scale Modeling (ESM) paradigm trains on vast sequence alignments to capture deep evolutionary constraints. This enables a model pretrained primarily on mammalian genomes to generalize meaningful predictions to non-model organisms through shared regulatory syntax.

A self-supervised pretraining objective where random input tokens are masked and the model learns to predict the original nucleotides from bidirectional context. This forces the model to learn deep regulatory grammar by understanding dependencies between upstream and downstream elements. Key aspects:

• Randomly masks 15-20% of input tokens during training
• Replaces masked tokens with a special [MASK] token or random nucleotide
• Loss is computed only on the masked positions
• Enables learning of promoter motifs, splice sites, and enhancer syntax without labeled data
• Adapted from BERT's original formulation for natural language to DNA k-mer sequences

A method that segments a nucleotide sequence into overlapping subsequences of fixed length k, converting raw DNA into discrete tokens. This bridges the gap between continuous biological sequences and discrete language model inputs. Common configurations:

• k=6 (hexamers) captures dinucleotide interactions and local motifs
• k=4 (tetramers) provides a vocabulary of 256 tokens for compact representation
• k=3 (trinucleotides) encodes codon-level information for coding regions
• Overlapping stride controls token density and sequence coverage
• Vocabulary size grows as 4^k, requiring balance between granularity and model complexity

A core Transformer component that computes a weighted representation of every position by comparing all pairs of positions simultaneously. This enables genomic models to capture long-range dependencies between distal regulatory elements. Biological relevance:

• Enhancer-promoter interactions spanning hundreds of kilobases
• Coordinated binding of transcription factors at composite motifs
• Chromatin looping and 3D genome architecture signals
• Splice site recognition requiring exon-intron boundary context
• The quadratic O(n²) complexity of standard attention drives research into efficient alternatives like FlashAttention and Hyena

Techniques used during self-supervised pretraining that deliberately introduce noise into input DNA sequences to force the model to learn robust, denoised representations. Common corruption approaches:

• Nucleotide masking: Replace individual bases with [MASK] tokens
• Span masking: Mask contiguous blocks of nucleotides to learn regional context
• Nucleotide substitution: Replace bases with random alternatives to simulate natural variation
• Sequence shuffling: Reorder local segments to learn positional dependencies
• Reverse complement injection: Present both strands to enforce strand-symmetric learning
• These strategies prevent the model from memorizing sequence patterns and encourage learning of fundamental biological grammar

A metric derived from a language model's cross-entropy loss that quantifies how surprised the model is by a given sequence. In genomics, lower perplexity indicates the sequence conforms to learned patterns of natural DNA. Applications:

• Evolutionary constraint detection: Conserved regions show low perplexity across species
• Variant effect prediction: Pathogenic mutations increase perplexity relative to reference
• Splice site validation: Canonical splice motifs exhibit lower perplexity than cryptic sites
• Sequence quality assessment: Anomalous or artifactual sequences produce high perplexity scores
• Perplexity provides a continuous, unsupervised measure of sequence functionality without task-specific training

A data augmentation strategy that presents both strands of a DNA sequence during training, enforcing the model to learn strand-symmetric representations. This reflects the double-helical nature of DNA where regulatory elements can function on either strand. Implementation details:

• Each training sequence is randomly presented in forward or reverse complement orientation
• The model learns that a promoter on the plus strand is equivalent to its reverse complement on the minus strand
• Prevents strand bias in predictions for tasks like transcription factor binding
• Critical for models processing unstranded sequencing data (e.g., ChIP-seq, ATAC-seq)
• Can be applied at the token level or as a model architecture constraint