Genomic Language Model (gLM)

gLMs leverage masked language modeling (MLM) to learn the statistical grammar of DNA without requiring labeled data. During pre-training, random nucleotide tokens are masked, and the model learns to predict them from surrounding bidirectional context. This forces the model to internalize promoter structures, splice junctions, and transcription factor binding motifs as emergent properties of sequence context. Models like DNABERT and the Nucleotide Transformer pre-train on reference genomes spanning the tree of life, producing embeddings that transfer to downstream tasks with minimal fine-tuning.

Unlike natural language, DNA lacks explicit word boundaries. gLMs employ specialized tokenization to segment the genome into learnable units:

• Overlapping k-mers: DNABERT uses 6-mers with stride 1, creating a vocabulary of 4,096 tokens that capture local sequence motifs
• Byte-Pair Encoding (BPE): Iteratively merges frequent nucleotide pairs into subword units, balancing vocabulary size with context length
• Codon tokenization: Segments coding sequences into non-overlapping triplets aligned with the genetic code
• Canonical k-mers: Collapses strand-specific representations by selecting the lexicographically smaller of a k-mer and its reverse complement

Regulatory elements like enhancers can act on promoters hundreds of kilobases away through chromatin looping. Standard transformers with quadratic self-attention cannot process such long sequences. gLMs address this through:

• Sparse attention mechanisms (BigBird, Longformer) that restrict each token to attend to a subset of others
• HyenaDNA's implicit long convolutions, replacing attention entirely to process up to 1 million tokens
• Enformer's hybrid architecture, combining convolutional layers for local motifs with transformer blocks for distal interactions across 200 kb input windows
• Rotary Position Embedding (RoPE) for length extrapolation beyond training sequence lengths

DNA is double-stranded, and regulatory elements can function on either strand. gLMs enforce biological consistency through reverse complement augmentation and canonical encoding. During training, both the forward sequence and its reverse complement are presented, doubling the dataset and teaching the model strand invariance. Some architectures use Siamese networks with shared weights to process both strands simultaneously, while others employ contrastive learning to pull embeddings of complementary strands together in latent space. Allele-specific embeddings further distinguish maternal and paternal haplotypes for cis-regulatory analysis.

Regulatory syntax is deeply conserved across evolution. gLMs pre-trained on diverse species learn species-agnostic embeddings that capture universal genomic grammar. The Nucleotide Transformer trains on genomes from human to yeast, enabling zero-shot transfer to organisms with limited annotations. Cross-species fine-tuning adapts a human-pre-trained model to mouse or zebrafish by leveraging conserved promoter architectures and transcription factor binding logic. Pangenome representations extend this further by encoding population-level structural variation into graph-based reference structures rather than single linear genomes.

Full fine-tuning of billion-parameter gLMs is computationally prohibitive for most research labs. Low-Rank Adaptation (LoRA) freezes pre-trained weights and injects trainable low-rank matrices into attention layers, reducing memory footprint by over 90% while maintaining performance on downstream tasks like variant effect prediction and chromatin accessibility inference. Adapter layers and prompt tuning offer alternative lightweight adaptation strategies. These methods enable a single pre-trained gLM to serve dozens of specialized assays without storing full model copies.

The dominant self-supervised pre-training objective adapted for genomic foundation models. The process randomly masks a percentage of input tokens (e.g., k-mers) and trains the model to predict the original nucleotide from the bidirectional context. This forces the model to learn the regulatory grammar and dependencies between distal genomic elements without requiring labeled data. Models like DNABERT use this to understand promoter-enhancer interactions.

A foundational genomic language model that adapts the BERT architecture for DNA. It tokenizes sequences into overlapping k-mers and pre-trains via masked language modeling on the human reference genome. Key capabilities include:

• Generating context-aware nucleotide embeddings
• Predicting promoters and splice sites
• Identifying transcription factor binding sites It established the paradigm of treating genomic sequence analysis as a natural language processing task.

A suite of genomic foundation models trained on diverse genomes across the tree of life. Unlike models restricted to the human genome, NT uses a standard transformer encoder to produce species-agnostic sequence embeddings. This cross-species pre-training enables robust transfer learning, allowing the model to identify conserved regulatory elements and predict variant effects in non-model organisms with limited annotations.

A genomic language model that replaces the quadratic self-attention mechanism with a sub-quadratic implicit long convolution operator. This architectural innovation enables the processing of ultra-long DNA sequences up to 1 million tokens in length, dramatically expanding the receptive field beyond the typical 512–2048 token limit of standard transformers. It captures long-range regulatory interactions across entire genomic loci without approximation.

A data compression algorithm repurposed for genomic tokenization. BPE iteratively merges the most frequent adjacent nucleotide pairs into subword units, building a vocabulary that balances sequence granularity with context length. Unlike fixed k-mer tokenization, BPE adapts to the statistical properties of the training corpus, creating shorter tokenizations for repetitive regions and longer tokens for unique sequences, optimizing the model's effective context window.

A deep learning architecture that combines convolutional layers with transformer self-attention to predict gene expression and epigenetic tracks directly from 200,000 base-pair DNA sequences. It explicitly models long-range regulatory interactions up to 100 kb away, dramatically outperforming previous methods on enhancer-gene pair prediction. Enformer demonstrates the power of gLMs to map from sequence to function.