Sequence Log-Likelihood

The model decomposes the joint probability of a sequence using the chain rule: log P(x) = Σ log P(xᵢ | x<ᵢ). Each nucleotide is predicted based solely on its preceding context. A higher log-likelihood indicates the sequence is a better fit to the training distribution of natural DNA.

• Unidirectional context: Only left-to-right (5' to 3') dependencies are modeled
• Token-level scoring: Probability is computed incrementally at each position
• Normalization: Log-space computation prevents numerical underflow for long sequences

Log-likelihood serves as a direct measure of evolutionary constraint. Functional genomic elements under purifying selection exhibit higher likelihood scores because they match the patterns the model learned from conserved regions across the genome.

• Coding exons typically score higher than intergenic regions
• Promoters and enhancers show elevated likelihood due to motif conservation
• Sharp drops in likelihood often indicate loss-of-function mutations

The functional impact of a genetic variant is quantified as the log-likelihood ratio (LLR) between the reference and alternate alleles: LLR = log P(x_ref) - log P(x_alt). A positive LLR indicates the variant disrupts the learned sequence patterns and is likely deleterious.

• Zero-shot capability: No labeled variant data required
• Context-aware: The model considers flanking sequence context
• Strand symmetry: Scores are typically averaged across both DNA strands

Perplexity is the exponentiated average negative log-likelihood: PPL = exp(-1/N Σ log P(xᵢ)). It represents the model's effective branching factor—how many equally likely choices the model faces at each position. Lower perplexity indicates stronger predictive confidence.

• Intuitive scale: Perplexity of 4 means the model is as uncertain as a uniform choice among 4 nucleotides
• Length normalization: Enables comparison across sequences of different lengths
• Benchmark standard: Primary metric for evaluating genomic language model performance

Log-likelihood enables computational saturation mutagenesis—systematically substituting every position with all possible nucleotides and measuring the likelihood change. Positions where substitutions cause large likelihood drops are predicted to be functionally critical.

• Nucleotide resolution: Identifies individual base pairs critical for function
• Scalable screening: Computationally evaluates all possible single-nucleotide variants
• Motif discovery: Clusters of sensitive positions often correspond to transcription factor binding sites

Sequences with abnormally low log-likelihood relative to a background distribution are flagged as statistical outliers. This principle is applied to detect sequencing artifacts, contamination, horizontal gene transfer events, and pathogenic insertions.

• Quality control: Identifies low-quality or misassembled genomic regions
• Pathogen detection: Foreign DNA often deviates from host genomic patterns
• Synthetic construct identification: Engineered sequences may violate natural k-mer frequencies

The probabilistic framework underlying sequence log-likelihood computation. An autoregressive model factorizes the joint probability of a sequence into a product of conditional probabilities, predicting each nucleotide given all preceding context.

• Chain Rule: P(x₁, x₂, ..., xₙ) = ∏ P(xᵢ | x₁, ..., xᵢ₋₁)
• Directionality: Typically processes DNA from 5' to 3', mirroring biological transcription
• Log Transformation: Summing log-probabilities prevents numerical underflow for long sequences
• Training Objective: Minimizes negative log-likelihood across a genomic corpus

A direct derivative of sequence log-likelihood that measures how 'surprised' a model is by a given DNA sequence. Lower perplexity indicates the sequence conforms to learned genomic grammar.

• Computation: Perplexity = exp(cross-entropy loss) = exp(-avg log-likelihood per token)
• Interpretation: A perplexity of 2 means the model is as uncertain as a fair coin flip at each position
• Constraint Detection: Evolutionarily conserved regions exhibit significantly lower perplexity than neutrally evolving DNA
• Model Comparison: Enables direct benchmarking between genomic language models on held-out sequences

The practical application of sequence log-likelihood for quantifying the functional impact of genetic variants. Computed as the log-likelihood ratio between reference and alternate alleles.

• Formula: Score = log P(seq_alt | model) - log P(seq_ref | model)
• Negative Scores: Indicate the variant reduces sequence likelihood, suggesting a disruptive or pathogenic effect
• Zero-Shot Capability: Genomic language models can score variants without training on labeled pathogenicity data
• Context Window: The surrounding sequence context provided to the model critically influences score accuracy

A systematic computational technique that leverages sequence log-likelihood to identify functionally critical nucleotides. By introducing every possible single-nucleotide substitution at each position and measuring the change in model likelihood, researchers can map regulatory elements.

• Saturation Scan: Evaluates all 3 alternate bases at every position in a regulatory region
• Likelihood Delta: Large drops in log-likelihood pinpoint nucleotides essential for protein binding or enhancer activity
• Attention Correlation: Likelihood-sensitive positions often align with high self-attention weights in Transformer models
• Experimental Validation: Predictions guide targeted mutagenesis experiments like massively parallel reporter assays (MPRAs)

The capability of genomic language models to predict variant pathogenicity using only sequence log-likelihood differences, without supervised fine-tuning on labeled clinical datasets.

• Mechanism: Relies entirely on the model's pretrained understanding of natural sequence distribution
• Evolutionary Proxy: The model implicitly learns evolutionary constraints from patterns in training data
• Clinical Relevance: Achieves competitive performance with dedicated pathogenicity predictors like CADD and PolyPhen-2
• Limitation: Struggles with gain-of-function variants that create novel regulatory elements not penalized by likelihood reduction

The dynamic nucleotide embeddings produced by genomic language models that enable log-likelihood computation. Unlike static k-mer embeddings, these vectors change based on surrounding sequence context, capturing regulatory syntax.

• Context Dependency: The representation of 'ACGT' differs when flanked by a promoter versus an intron
• Attention-Based: Self-attention layers mix information across positions to build contextualized vectors
• Logit Generation: Final layer representations are projected to nucleotide vocabulary probabilities for likelihood calculation
• Probing Tasks: Contextualized embeddings can be decoded to predict chromatin state, TF binding, and splicing