Masked Language Modeling (MLM)

Unlike autoregressive models that only see preceding tokens, MLM leverages the full bidirectional context surrounding a masked nucleotide. This is critical for genomics because regulatory elements like enhancers and promoters often interact across vast linear distances in both directions. The model learns that a nucleotide's identity is constrained by both upstream and downstream sequence context, capturing the non-linear syntax of gene regulation.

During pretraining, a random subset of input tokens is corrupted. The standard strategy involves:

• Masking 15% of tokens: A fixed proportion of k-mers or nucleotides are selected.
• Replacement protocol: 80% are replaced with a special [MASK] token, 10% with a random token, and 10% left unchanged.
• Objective: The model must predict the original nucleotide at each masked position, forcing it to learn a deep internal representation of regulatory syntax and evolutionary constraints.

Standard MLM masks random individual tokens, but genomic sequences contain long-range correlations and repetitive elements. To prevent the model from cheating by simply copying adjacent identical k-mers, genomic MLM often employs contiguous span masking. This strategy masks entire blocks of consecutive tokens, forcing the model to rely on distal regulatory signals—such as a faraway enhancer—to reconstruct the missing sequence, thereby learning biologically meaningful long-range dependencies.

The MLM objective compels the model to internalize the complex grammar of the genome. By predicting masked nucleotides, the model learns:

• Transcription factor binding motifs: Short, conserved sequence patterns.
• Splice sites: Donor and acceptor signals for RNA processing.
• Chromatin accessibility: Sequence determinants of open vs. closed chromatin. This learned representation serves as a powerful foundation for downstream tasks like variant effect prediction and promoter identification.

A powerful emergent property of MLM-trained genomic models is zero-shot variant effect prediction. By comparing the model's predicted probability for the reference allele versus an alternate allele at a masked position, one can compute a variant effect score. A significant drop in probability indicates a potentially deleterious mutation. This allows the model to assess the functional impact of genetic variants without any supervised fine-tuning on labeled clinical data.

While autoregressive models (like GPT) predict the next token, MLM (like BERT) reconstructs corrupted tokens. For genomics, MLM is often preferred because:

• Bidirectional context is essential for understanding regulatory elements that are flanked by signals on both sides.
• Denoising objective directly mirrors the task of identifying functional elements that are conserved despite mutational noise.
• Representation learning yields superior embeddings for classification tasks compared to unidirectional approaches.

During training, a fraction of input tokens—typically 15% of k-mers—are randomly selected for corruption. Of these selected tokens, 80% are replaced with a special [MASK] token, 10% are replaced with a random nucleotide token, and 10% are left unchanged. The model's objective is to predict the original nucleotide identity at each masked position using bidirectional context from both upstream and downstream sequence. This forces the model to learn dependencies in both directions, unlike autoregressive models that only see preceding context. The final loss is computed only on the masked positions, not the entire sequence.

Before masking, raw DNA sequences must be tokenized into discrete units. The two dominant strategies are:

• K-mer Tokenization: Overlapping subsequences of fixed length k (e.g., 3, 4, 5, or 6). A 6-mer vocabulary has 4^6 = 4,096 tokens. This is the approach used by DNABERT.
• Byte-Pair Encoding (BPE): Iteratively merges the most frequent nucleotide pairs to build a subword vocabulary. This handles open-vocabulary challenges and is used by models like DNABERT-2. Tokenization converts a string like ATCGAT into a sequence of integers that the Transformer can process.

The defining advantage of MLM over autoregressive models is bidirectionality. A masked nucleotide at position i receives gradient signals from both the 5' and 3' flanking sequence. This is critical for genomics because:

• Transcription factor binding sites are palindromic or have flanking sequence preferences.
• Splice sites require both upstream (acceptor) and downstream (donor) context.
• Enhancer-promoter loops involve distal elements acting in cis from either direction. The model learns that regulatory grammar is not strictly linear but context-dependent in both directions.

Standard random masking may not be optimal for DNA. Advanced strategies include:

• Contiguous Span Masking: Masking consecutive k-mers rather than isolated tokens, forcing the model to reconstruct longer regulatory motifs. Used in genomic masked autoencoders.
• Stride-Based Masking: Masking every n-th token to ensure uniform coverage across the sequence.
• Motif-Aware Masking: Prioritizing masking of known regulatory elements (promoters, enhancers) to focus learning on functionally important regions.
• Reverse Complement Consistency: Ensuring that if a forward-strand token is masked, its reverse complement on the opposite strand is also masked, enforcing strand symmetry.

A powerful emergent capability of MLM-pretrained genomic models is zero-shot variant effect prediction. By comparing the model's predicted probability of the reference allele versus an alternate allele at a masked position, the log-likelihood ratio serves as a pathogenicity score. This requires no labeled variant data—only the pretrained model's internal sequence understanding. Models like EVE and gLM have shown that this unsupervised approach rivals supervised methods like CADD and PolyPhen-2 for identifying disease-causing mutations.

The initial phase of training a DNA language model on massive, unlabeled genomic corpora using self-supervised objectives like MLM to learn universal sequence representations. This process allows the model to internalize regulatory syntax, promoter structures, and splice sites without manual annotation. The resulting weights serve as a starting point for fine-tuning on specific downstream tasks such as variant effect prediction.

Techniques used during self-supervised pretraining that deliberately introduce noise into input DNA sequences to force the model to learn robust, denoised representations. In standard MLM, 15% of tokens are masked: 80% replaced with a [MASK] token, 10% with a random nucleotide, and 10% left unchanged. This prevents the model from simply copying the input and forces learning of deep regulatory grammar.

A self-supervised architecture that masks large contiguous spans of a DNA sequence—rather than random individual tokens—and trains a model to reconstruct the missing nucleotides. This approach forces the model to learn higher-order chromatin architecture and long-range dependencies. Unlike standard MLM, the encoder only processes visible tokens, dramatically reducing compute cost for very long sequences.

A data augmentation strategy that presents both strands of a DNA sequence during MLM training, enforcing the model to learn strand-symmetric representations. Since DNA is double-stranded and regulatory elements can reside on either strand, this augmentation prevents the model from learning strand-specific artifacts. The model must predict the correct nucleotide regardless of which strand the masked token appears on.

Dynamic nucleotide embeddings generated by a genomic language model where the vector for a given k-mer changes depending on its surrounding sequence context. Unlike static embeddings, these representations capture regulatory syntax—the same motif may have different meanings in a promoter versus an enhancer. MLM is the key objective that forces the model to build these context-aware representations.