Masked Language Modeling (MLM)

Unlike autoregressive models that process DNA left-to-right, MLM enables models to learn from both upstream and downstream sequence context simultaneously. When a nucleotide is masked, the model attends to flanking regions in both directions, capturing regulatory syntax like enhancer-promoter interactions that span thousands of base pairs. This bidirectional understanding is critical for predicting transcription factor binding sites and splice junctions where context on both sides determines function.

During pre-training, 15% of input tokens are randomly selected for masking. Of these selected tokens:

• 80% are replaced with a special [MASK] token
• 10% are replaced with a random nucleotide
• 10% remain unchanged This stochastic strategy forces the model to learn robust, context-dependent representations rather than simply memorizing nucleotide frequencies, preventing overfitting to the masking pattern itself.

Genomic MLM models like DNABERT tokenize sequences into overlapping k-mers (typically k=3 to 6) rather than single nucleotides. This captures local sequence motifs such as dinucleotide frequencies and short regulatory elements within each token. The overlapping stride ensures that each nucleotide appears in multiple tokens, providing redundancy that improves prediction accuracy when the model reconstructs masked positions from surrounding k-mer embeddings.

MLM models are pre-trained on unlabeled genomic corpora such as the human reference genome (GRCh38) or multi-species collections. The model learns the statistical grammar of DNA—promoter architecture, exon-intron boundaries, and repetitive element distributions—without requiring functional annotations. This produces transferable embeddings that can be fine-tuned for downstream tasks like variant effect prediction or chromatin state classification with limited labeled data.

After MLM pre-training, the model is adapted to specific genomic prediction tasks through supervised fine-tuning. The pre-trained encoder weights are loaded, and a task-specific classification or regression head is added. Common downstream applications include:

• Variant pathogenicity prediction from surrounding sequence context
• Promoter strength estimation for synthetic biology
• Transcription factor binding site identification The bidirectional representations learned during MLM provide a rich initialization that dramatically reduces the labeled data required.

Genomic MLM implementations often incorporate strand-awareness by ensuring that a masked k-mer and its reverse complement receive consistent predictions. During training, sequences are randomly presented in forward or reverse-complement orientation, and the loss function penalizes strand-inconsistent reconstructions. This enforces the biological symmetry of double-stranded DNA, improving performance on tasks like motif discovery where binding sites function regardless of strand orientation.

A foundational genomic language model that adapts the BERT architecture by tokenizing DNA sequences into overlapping k-mers and pre-training via masked language modeling on the human reference genome. DNABERT learns context-aware nucleotide embeddings that capture regulatory syntax and promoter structures, enabling state-of-the-art performance on downstream tasks like splice site prediction and transcription factor binding site identification.

A training paradigm where the model generates its own supervisory signal from unlabeled data, eliminating the need for expensive manual annotation. In genomic MLM, the objective is to predict masked nucleotides from surrounding sequence context. This forces the model to learn intrinsic biological patterns—such as codon usage bias, splice junctions, and regulatory motifs—before fine-tuning on specific prediction tasks with limited labeled data.

The method of segmenting raw nucleotide sequences into discrete units that the model can process. Common approaches include: k-mer tokenization (overlapping substrings of length k), non-overlapping k-mers, Byte-Pair Encoding (BPE) which merges frequent adjacent nucleotide pairs, and codon tokenization that aligns with the genetic code's triplet structure. Tokenization directly impacts vocabulary size, sequence length, and the model's ability to learn meaningful biological representations.

Unlike autoregressive models that only attend to previous tokens, MLM enables each token to incorporate information from both upstream and downstream sequence context simultaneously. This is critical for genomics because regulatory elements like enhancers can act from either direction, and splice sites depend on both the 5' donor and 3' acceptor motifs. Bidirectional attention captures the full cis-regulatory grammar of DNA.

The specific protocol for selecting which tokens to hide during pre-training. Standard BERT masks 15% of tokens randomly, but genomic adaptations may use: contiguous span masking to hide entire motifs, dynamic masking that changes patterns each epoch, or biologically-informed masking that targets known functional elements. The masking rate and pattern significantly influence whether the model learns local nucleotide preferences or long-range regulatory dependencies.

An alternative self-supervised objective that trains an encoder to maximize mutual information between latent representations of distant sequence patches. Unlike MLM which reconstructs masked tokens directly, CPC uses a contrastive loss to distinguish future context from random negative samples. In genomics, CPC excels at learning slowly varying features like chromatin states and topological domains that span tens of thousands of base pairs.