Contextualized Sequence Representations

A specific k-mer (e.g., 'TATAAA') will have a different vector representation depending on whether it appears in a promoter region, an intron, or a coding exon. The model dynamically re-weights the embedding based on surrounding regulatory grammar.

• Eliminates the polysemy problem inherent in static embeddings
• Captures the functional syntax of the genome
• Enables distinction between identical motifs with different biological roles

Contextualized models use self-attention mechanisms to directly model interactions between distal genomic elements, such as enhancers and their target promoters, which may be separated by hundreds of kilobases.

• Weights the influence of every position on every other position
• Learns complex 3D chromatin interaction logic from linear sequence
• Critical for understanding gene regulation in non-coding regions

Unlike autoregressive models that only see left-to-right context, architectures like BERT-based DNA models build representations by attending to nucleotides on both flanks simultaneously.

• Essential for identifying splice sites and regulatory elements
• Mirrors the biological reality of bidirectional transcription factor binding
• Produces richer representations for downstream classification tasks

Sophisticated models enforce reverse complement invariance, ensuring that a sequence and its reverse complement produce equivalent representations, consistent with the double-helical nature of DNA.

• Achieved through data augmentation or architectural constraints
• Prevents the model from learning strand-specific artifacts
• Improves generalization across forward and reverse strand annotations

Through self-supervised pretraining on massive genomic corpora, these models learn implicit representations of biological grammar—including transcription factor binding motifs, splice junctions, and chromatin states—without explicit supervision.

• Attention heads often recover known position weight matrices
• Internal representations correlate with experimental epigenomic tracks
• Enables zero-shot prediction of functional elements

Modern architectures using Rotary Position Embeddings (RoPE) or state-space models produce representations that generalize to sequence lengths far exceeding those seen during training.

• Enables processing of entire gene loci or megabase-scale regions
• Avoids the context-window limitations of fixed positional encodings
• Critical for analyzing structural variants and large genomic rearrangements

The computational engine driving contextualization. Self-attention computes a weighted representation for every nucleotide position by comparing it against all other positions in the sequence simultaneously.

• Calculates Query (Q), Key (K), and Value (V) vectors for each token.
• Attention weights are derived from the compatibility (dot product) between Q and K.
• Enables the model to directly connect a distal enhancer to its target promoter, regardless of linear distance.
• This is why the embedding for a 'T' in a coding exon differs from a 'T' in a repetitive element.

The dominant self-supervised pretraining objective that teaches the model contextual syntax. Random tokens in an input sequence are hidden, and the model must predict the original nucleotide from the bidirectional context.

• Forces the model to learn regulatory grammar, not just nucleotide frequency.
• A masked splice donor site must be predicted using surrounding exonic and intronic signals.
• This objective directly produces contextualized representations because the model's prediction depends entirely on the unmasked flanking sequence.

The process of segmenting raw DNA into overlapping fixed-length subsequences before embedding. A sequence ATCGAT with k=3 becomes ATC, TCG, CGA, GAT.

• Converts a 4-letter alphabet into a vocabulary of 4^k possible tokens.
• Provides a local sequence motif context that single nucleotides lack.
• The embedding for a specific k-mer is the initial non-contextualized vector that the Transformer layers will subsequently refine based on surrounding k-mers.

The biological relationships between genomic elements separated by vast linear distances—often tens of thousands to millions of base pairs. Capturing these is the primary challenge for genomic models.

• Classic example: an enhancer regulating a promoter located 50kb away.
• Standard CNNs fail here due to limited receptive fields.
• Contextualized representations solve this by encoding the functional relationship between the two distant elements into their respective vectors, making them 'aware' of each other.

The initial, computationally intensive phase where a DNA language model learns universal sequence representations from massive, unlabeled genomic corpora. This is where contextualization is learned.

• The model ingests raw assemblies from diverse species without specific task labels.
• Through objectives like MLM, it learns to distinguish promoters from enhancers and coding from non-coding sequence purely from statistical structure.
• The resulting weights form a genomic foundation model that can be fine-tuned for specific tasks like variant effect prediction.

A computational technique that directly leverages contextualized representations to identify functional nucleotides. A virtual mutation is introduced, and the change in the model's internal representations or output predictions is measured.

• A large delta in the embedding of a surrounding regulatory region indicates the mutated base was highly influential.
• This reveals transcription factor binding sites and critical splice junctions without wet-lab experiments.
• The method works because the mutation disrupts the learned contextual syntax that the model has internalized.