Masked Language Modeling (MLM)

During pre-training, a random subset of input tokens—typically 15% of nucleotides or k-mers—is selected for corruption. Of these selected positions, 80% are replaced with a special [MASK] token, 10% are replaced with a random token, and 10% remain unchanged. This strategy prevents the model from simply learning to detect mask tokens and forces it to build a robust internal representation of the sequence distribution. In genomic contexts, masking is often applied uniformly across the sequence, but advanced strategies use span-based masking to corrupt contiguous regions, better mimicking the contiguous nature of regulatory elements like enhancers and promoters.

Unlike autoregressive models that process sequences left-to-right, MLM enables the model to attend to both upstream and downstream context simultaneously when predicting a masked token. This bidirectionality is critical for genomics because regulatory elements—such as transcription factor binding sites—are defined by flanking sequence context on both sides. A promoter's function depends equally on upstream TATA boxes and downstream transcription start sites. By conditioning on the full surrounding context, the model learns position-independent regulatory syntax that captures the true grammar of genomic elements.

The training objective minimizes the cross-entropy loss between the model's predicted nucleotide probability distribution and the true identity of the masked token. For a vocabulary of 4 nucleotides (A, C, G, T) or thousands of k-mers, the model outputs a softmax distribution over the vocabulary at each masked position. The loss is computed only on the masked positions, not on the unmasked tokens. This sparse loss signal forces the model to extract maximum information from limited supervision, leading to emergent capabilities like zero-shot variant effect prediction, where the change in predicted probability between reference and alternate alleles correlates with functional impact.

Through MLM pre-training on diverse genomes, the model implicitly learns evolutionary conservation patterns without explicit alignment data. Positions that are highly conserved across species exhibit low entropy in the model's predicted distribution—the model is confident about which nucleotide belongs there. Conversely, variable positions show high entropy. This emergent property enables the model to distinguish purifying selection from neutral drift. The learned conservation scores from attention heads and embedding representations correlate strongly with phyloP and GERP++ scores, traditional measures of evolutionary constraint derived from multiple sequence alignments.

Each token in an MLM-trained model receives a context-dependent embedding that differs based on surrounding sequence, unlike static embeddings from models like word2vec. The same k-mer appearing in a coding exon versus an intergenic region will have different vector representations because the model integrates flanking regulatory signals. These contextualized embeddings serve as transferable features for downstream tasks: a linear classifier trained on top of frozen embeddings can predict chromatin accessibility, transcription factor binding, and splice sites with performance approaching fully supervised models, enabling effective fine-tuning with limited labeled data.

Standard token-level masking is often extended to contiguous span masking for genomic applications. Instead of masking individual k-mers independently, entire spans of 5-20 tokens are masked simultaneously, forcing the model to reconstruct longer regulatory sequences. This variant, sometimes called SpanBERT-style masking, is particularly effective for learning enhancer syntax and long-range promoter-enhancer interactions. The model must infer the identity of entire regulatory modules from distal context, learning the compositional grammar of transcription factor binding site clusters rather than isolated motif recognition.

The overarching learning paradigm where a model generates its own supervisory signal from unlabeled data. In MLM, the masked tokens themselves serve as the labels, eliminating the need for expensive, manually annotated genomic datasets. This enables training on millions of unlabeled genomes across diverse species, allowing the model to internalize fundamental biological sequence constraints before any task-specific fine-tuning occurs.

The method of segmenting raw nucleotide sequences into discrete tokens directly impacts MLM effectiveness. Common approaches include:

• Single nucleotide tokenization: Vocabulary of 4-5 tokens (A, T, C, G, N), preserving single-base resolution
• K-mer tokenization: Overlapping subsequences of length k (e.g., 6-mers), capturing local motif patterns
• Byte-Pair Encoding (BPE): Data-driven subword tokenization that balances vocabulary size against sequence compression The choice affects the masking granularity and the model's ability to learn both local and long-range dependencies.

The proportion of tokens masked during pre-training is a critical hyperparameter. In genomic MLM, typical masking rates range from 15% to 25%, higher than NLP due to the informational redundancy in DNA. Advanced strategies include:

• Span masking: Masking contiguous blocks of nucleotides to force learning of longer-range dependencies
• Dynamic masking: Generating new mask patterns each epoch to prevent memorization
• Entity-aware masking: Masking entire functional elements like transcription factor binding motifs to teach regulatory syntax The masking strategy determines whether the model learns local nucleotide preferences or higher-order chromatin grammar.

The primary output of an MLM-trained genomic model is a dense vector representation for each nucleotide position that captures its functional context. Unlike static one-hot encodings, these embeddings encode:

• Evolutionary conservation signals indicating functional constraint
• Regulatory potential for transcription factor binding
• Structural context related to chromatin accessibility and 3D genome organization These embeddings serve as transferable features for downstream tasks like variant effect prediction and enhancer identification without task-specific architecture changes.

MLM pre-training on diverse genomes enables remarkable cross-species generalization. A model trained primarily on human and mouse genomes can accurately predict regulatory elements in zebrafish or Arabidopsis with minimal fine-tuning. This works because:

• Fundamental transcription factor binding preferences are conserved across eukaryotes
• Chromatin organization principles share deep evolutionary roots
• The model learns universal sequence-structure relationships rather than species-specific motifs This transferability dramatically reduces the labeled data required for non-model organisms.

A powerful emergent capability of MLM-trained genomic models is the ability to score mutations without any supervised training on labeled variant data. By computing the log-likelihood difference between the reference and alternate allele in their sequence context, the model quantifies functional impact. This approach:

• Correlates strongly with ClinVar pathogenicity annotations
• Identifies deleterious non-coding variants in regulatory elements
• Provides a continuous effect score for every possible single-nucleotide change The zero-shot nature eliminates the circularity and ascertainment bias inherent in training on known pathogenic variants.