Sequence Corruption Strategies

The foundational corruption strategy adapted from natural language processing. A random subset of input tokens is replaced with a special [MASK] token, and the model learns to predict the original nucleotides from bidirectional context.

• Corruption rate: Typically 15% of tokens are masked, with 80% replaced by [MASK], 10% by random tokens, and 10% left unchanged
• Bidirectional context: Forces the model to integrate upstream and downstream regulatory signals
• Genomic adaptation: Masks k-mer tokens rather than individual nucleotides, preserving local motif integrity
• Key insight: The model cannot simply memorize motifs; it must learn the regulatory grammar that dictates which sequences appear in which contexts

Used extensively in DNABERT and other BERT-derived genomic architectures.

Instead of masking individual tokens, contiguous spans of the input sequence are corrupted. The model must reconstruct entire missing regions, learning higher-order structural dependencies.

• Span length: Ranges from short motifs to megabase-scale deletions in long-context models
• Genomic Masked Autoencoder: Masks large contiguous blocks (e.g., 50% of the sequence) and uses an asymmetric encoder-decoder to reconstruct the missing DNA
• Biological relevance: Mimics structural variants, insertions, and deletions found in real genomes
• Efficiency: The encoder only processes visible tokens, dramatically reducing compute for long sequences

This strategy teaches models to understand syntenic relationships and long-range regulatory logic that span tens of thousands of nucleotides.

Individual nucleotides or k-mer tokens are randomly replaced with alternatives, simulating point mutations. The model learns to be robust to natural genetic variation and sequencing errors.

• Substitution rate: Typically 5-20% of positions are altered
• Biological parallel: Mirrors single nucleotide polymorphisms (SNPs) and somatic mutations
• Denoising objective: The model must recover the original sequence, learning which substitutions are tolerated and which disrupt function
• Strand awareness: Often combined with reverse complement augmentation to enforce strand-symmetric representations

This corruption type directly contributes to a model's ability to perform zero-shot variant effect prediction, where the log-likelihood difference between reference and alternate alleles scores pathogenicity.

Tokens are randomly deleted from or inserted into the input sequence, forcing the model to learn frameshift-robust representations and the grammar of insertions and deletions (indels).

• Deletion rate: Random tokens are dropped, creating gaps the model must implicitly fill
• Insertion noise: Spurious tokens are added, teaching the model to ignore irrelevant signals
• Frameshift resilience: Critical for handling sequencing errors and real biological indels
• Alignment-free learning: The model learns to recognize functional elements even when the precise positional frame is disrupted

This strategy is particularly important for models that process raw sequencing reads or must generalize across species with different genome architectures.

A unidirectional corruption strategy where the model predicts each subsequent token based solely on preceding context. The corruption is implicit: the model's own uncertainty about the future acts as the training signal.

• Causal masking: Attention is restricted to previous positions only
• Sequence likelihood: The model learns a probability distribution over all possible next tokens
• Genomic application: Used for sequence likelihood estimation and synthetic DNA generation
• Perplexity scoring: The model's surprise at each position becomes a measure of evolutionary constraint

Unlike MLM, autoregressive modeling naturally handles variable-length sequences and is the foundation for generative genomic models that can sample novel regulatory elements.

Modern genomic foundation models combine multiple corruption strategies simultaneously during pretraining, learning richer representations than any single objective alone.

• Joint objectives: MLM + autoregressive + contrastive losses are optimized together
• Dynamic scheduling: The corruption type and rate may vary across training steps
• Complementary signals: Masking teaches bidirectional context, while autoregressive modeling captures sequential dependencies
• Contrastive corruption: Positive pairs (e.g., overlapping genomic windows) are pulled together in embedding space while negative pairs are pushed apart

This multi-task approach produces contextualized sequence representations that transfer effectively across diverse downstream tasks, from promoter prediction to chromatin accessibility modeling.

The foundational objective adapted from natural language processing where random input tokens are masked and the model learns to predict the original nucleotides from bidirectional context.

• Mechanism: A percentage of k-mers are replaced with a [MASK] token.
• Genomic Adaptation: Forces the model to learn regulatory grammar and dependencies between motifs.
• Key Distinction: Unlike text, genomic masks often target contiguous spans to prevent trivial inference from adjacent nucleotides.

A self-supervised architecture that masks large contiguous spans of a DNA sequence and trains a model to reconstruct the missing nucleotides.

• High Masking Ratio: Often masks 50-80% of the sequence to create a challenging reconstruction task.
• Representation Learning: Learns rich internal representations of regulatory grammar and long-range dependencies.
• Efficiency: The encoder processes only visible tokens, dramatically reducing compute for long sequences.

A data augmentation strategy that presents both strands of a DNA sequence during training, enforcing the model to learn strand-symmetric representations.

• Biological Reality: DNA is double-stranded; regulatory elements can function on either strand.
• Implementation: Each sequence is randomly presented in its forward or reverse-complement orientation.
• Outcome: Prevents the model from learning spurious strand-specific artifacts.

A computational technique that systematically introduces virtual mutations into a DNA sequence and measures the resulting change in model predictions.

• Purpose: Identifies nucleotides critical for regulatory function.
• Method: Every position is individually mutated to all three alternative bases.
• Output: A saturation mutagenesis map showing the predicted functional impact of each substitution.

The probability assigned to a genomic sequence by an autoregressive model, used as a measure of how well the sequence conforms to learned patterns of natural DNA.

• Constraint Scoring: Low likelihood indicates a sequence deviates from evolutionary norms.
• Pathogenicity Prediction: Variants that reduce likelihood are often deleterious.
• Corruption Link: Models trained with noise learn to assign higher likelihood to clean, functional sequences.

The computational or experimental process of evaluating the functional impact of every possible single-nucleotide substitution at a given locus.

• Computational Acceleration: Genomic language models can predict all variant effects in seconds.
• Application: Prioritizing variants in rare disease diagnosis.
• Corruption Connection: Pretraining with sequence corruption teaches models the 'grammar' needed to recognize disruptive mutations.