Perplexity Scoring

Perplexity is defined as the exponentiated cross-entropy loss: PPL = exp(L) where L is the average negative log-likelihood per token. A perplexity of k means the model is as uncertain as if it were choosing uniformly among k options. For genomic models, lower perplexity on held-out sequences indicates the model has learned the underlying regulatory grammar and evolutionary constraints of the genome.

Genomic regions under purifying selection exhibit lower perplexity because their sequences conform to learned functional patterns. Key applications include:

• Coding exons: Consistently low perplexity due to codon bias and amino acid constraints
• Regulatory elements: Promoters and enhancers show reduced perplexity reflecting transcription factor binding site grammar
• Ultraconserved elements: Near-zero perplexity across species, indicating extreme functional importance

Deviations from low perplexity in these regions often signal pathogenic variants.

The functional impact of a genetic variant can be scored by comparing the perplexity of the reference allele versus the alternate allele:

• Log-likelihood ratio: LLR = log(P(alt_seq)) - log(P(ref_seq))
• Negative LLR indicates the variant disrupts learned regulatory patterns
• This zero-shot approach requires no labeled training data, leveraging only the pretrained model's sequence understanding
• Models like DNABERT and HyenaDNA use this method for pathogenicity prediction without task-specific fine-tuning

Perplexity enables base-pair resolution functional annotation through systematic virtual mutation:

• Saturation mutagenesis: Every possible single-nucleotide substitution is introduced computationally
• The change in perplexity for each substitution reveals which positions are intolerant to change
• Positions with large perplexity increases are predicted to be functionally critical
• This technique identifies transcription factor binding sites, splice junctions, and regulatory motifs without experimental assays

Perplexity scores require careful calibration for comparative analysis:

• Tokenization matters: K-mer size and BPE vocabulary directly affect absolute perplexity values
• Model-specific baselines: Each architecture (DNABERT vs. HyenaDNA vs. Enformer) produces different perplexity distributions
• Species-specific models: A model trained on human genomes will assign higher perplexity to mouse sequences, reflecting genuine evolutionary distance
• Length normalization: Longer sequences naturally accumulate higher total loss; perplexity normalizes by token count for fair comparison

Perplexity has important constraints in genomic applications:

• Context window limits: Standard Transformers cannot evaluate dependencies beyond their maximum sequence length, missing long-range enhancer-promoter interactions
• Strand symmetry: Models must be trained with reverse complement augmentation to avoid strand-specific perplexity artifacts
• Repetitive elements: Low-complexity regions and tandem repeats can artificially inflate or deflate perplexity
• Causal vs. masked models: Autoregressive models compute forward perplexity while masked models use pseudo-perplexity, which are not directly comparable

The fundamental training objective and mathematical basis for perplexity. Cross-entropy quantifies the dissimilarity between the model's predicted probability distribution over the nucleotide vocabulary and the true distribution (the actual next token). For a genomic sequence, it measures the average negative log-likelihood of each token given its context. A lower cross-entropy indicates the model assigns high probability to the correct nucleotides, directly translating to a lower perplexity score. It is the raw loss value that perplexity exponentiates for a more intuitive scale.

The natural logarithm of the probability a model assigns to a full genomic sequence. For an autoregressive model, this is the sum of the log-probabilities of each nucleotide given all preceding ones. This value is the direct input for calculating perplexity (perplexity = exp(-average log-likelihood)). In variant effect prediction, the difference in log-likelihood between a reference and alternate allele sequence serves as a zero-shot pathogenicity score, quantifying how much a mutation disrupts the learned grammar of natural DNA.

An alternative metric to perplexity, especially common in genomic modeling, that measures the average number of bits needed to encode each nucleotide in a sequence. BPC is calculated as the cross-entropy loss divided by the natural logarithm of 2. It provides an information-theoretic interpretation: a BPC of 2.0 means the model requires 2 bits on average to predict each base (A, C, G, T). Lower BPC indicates a better model. Unlike perplexity, BPC is length-agnostic and directly comparable across models with different tokenization schemes.

A key application of perplexity scoring where a genomic language model predicts the functional impact of a genetic variant without any supervised fine-tuning on labeled pathogenic data. The process involves:

• Computing the log-likelihood of the reference sequence
• Computing the log-likelihood of the alternate (mutated) sequence
• Taking the difference as a variant effect score A large negative score indicates the mutation significantly surprises the model, suggesting it disrupts a conserved regulatory element or coding region. This leverages the model's pretrained understanding of evolutionary constraints.

A systematic computational technique that uses perplexity or model predictions to measure the regulatory importance of every nucleotide in a sequence. The process introduces every possible single-nucleotide substitution at each position and records the change in the model's output or sequence likelihood. Nucleotides that cause a large increase in perplexity (or decrease in predicted expression) when mutated are identified as critical for function. ISM produces high-resolution saliency maps of regulatory elements, revealing transcription factor binding sites and other functional motifs.

The use of perplexity and sequence likelihood to identify genomic regions under purifying selection. The principle is that functionally important DNA elements evolve slowly because mutations are deleterious and removed by natural selection. A genomic language model trained on extant sequences learns this constrained grammar. Regions where the model assigns high probability (low perplexity) and where mutations cause a large drop in likelihood are inferred to be under strong evolutionary constraint, serving as a computational proxy for functional importance without requiring multi-species alignments.