Autoregressive Genomic Modeling

The core architectural constraint of autoregressive models is the causal attention mask, which prevents each nucleotide from attending to future positions in the sequence. This triangular mask ensures the model learns a left-to-right factorization of the joint probability distribution P(x₁, x₂, ..., xₙ) = ∏ P(xᵢ | x₁, ..., xᵢ₋₁).

• Mechanism: Each position queries only keys from preceding positions
• Training: Teacher forcing with ground-truth previous tokens
• Inference: Sequential generation, one token at a time
• Contrast: Unlike bidirectional models (e.g., DNABERT), causal models cannot incorporate downstream context

This constraint makes autoregressive models natural sequence generators but limits their ability to capture regulatory elements that depend on both upstream and downstream sequence context.

Autoregressive models compute the exact log-likelihood of any DNA sequence by evaluating the product of conditional probabilities across all positions. This tractable likelihood serves as a powerful signal for biological discovery.

• Perplexity scoring: Lower perplexity indicates sequences that conform to learned genomic patterns
• Evolutionary constraint: Conserved regions exhibit lower perplexity under models trained on diverse genomes
• Pathogenicity prediction: Variants that increase perplexity are more likely to be deleterious
• Anomaly detection: Synthetic or engineered sequences can be identified by unusually high perplexity

The sequence log-likelihood provides a principled, unsupervised metric for quantifying how well any DNA segment fits the model's learned distribution of natural genomic sequences.

A defining capability of autoregressive genomic models is predicting the functional impact of genetic variants without any task-specific training. The variant effect score is computed as the log-likelihood ratio between the alternate and reference alleles.

Score = log P(sequence with alt allele) - log P(sequence with ref allele)

• Negative scores: Alternate allele is less likely under the model, suggesting a deleterious variant
• Positive scores: Alternate allele is more likely, suggesting a potential gain-of-function
• Near-zero scores: Neutral or benign variant

This approach has shown strong correlation with deep mutational scanning experiments and clinical pathogenicity databases, enabling variant prioritization directly from sequence context without labeled training data.

Autoregressive models are generative by design, capable of sampling novel DNA sequences from the learned distribution by iteratively predicting and sampling the next nucleotide. This enables the creation of artificial genomic elements with desired properties.

• Promoter design: Generate synthetic promoters with specified expression levels by conditioning on functional tags
• Enhancer engineering: Sample novel enhancer sequences that maintain cell-type specificity
• Data augmentation: Create realistic training sequences for downstream models when real data is scarce
• Privacy preservation: Generate synthetic genomes that retain population-level statistics without exposing individual data

The generation process uses temperature-controlled sampling and nucleus (top-p) sampling to balance sequence novelty with biological plausibility.

Autoregressive models enable systematic computational saturation mutagenesis, where every possible single-nucleotide substitution at a locus is evaluated by measuring the change in model likelihood. This identifies nucleotides critical for regulatory function.

• Process: For each position, substitute A→C/G/T and compute Δ log-likelihood
• Interpretation: Large negative Δ values indicate highly constrained positions
• Scale: A 1,000 bp region requires only 3,000 forward passes (3 alternate bases × 1,000 positions)
• Comparison: Far more efficient than bidirectional models, which require full forward passes for each substitution

This technique reveals transcription factor binding footprints, splice site boundaries, and regulatory grammar at nucleotide resolution without experimental assays.

The training objective for autoregressive genomic models is causal language modeling: minimize the cross-entropy loss between the predicted nucleotide distribution and the true next nucleotide at each position.

Loss = -Σᵢ log P(xᵢ | x₁, ..., xᵢ₋₁; θ)

• Teacher forcing: Ground-truth previous tokens are always provided, not model predictions
• Parallelization: Loss can be computed for all positions simultaneously during training
• Tokenization: Requires a genomic tokenizer (k-mer, BPE, or single-nucleotide) to convert DNA to discrete tokens
• Strand handling: Reverse complement augmentation ensures strand-symmetric learning

This objective directly optimizes the model's ability to capture local nucleotide dependencies and long-range regulatory syntax across genomic sequences.

Leverages the sequence log-likelihood assigned by an autoregressive model to assess the functional impact of genetic variants without task-specific training. By comparing the probability of the reference allele against the alternate allele, the model computes a variant effect score that correlates strongly with pathogenicity.

• Mechanism: The log-likelihood ratio quantifies how much a mutation disrupts the natural sequence grammar learned during pretraining.
• Application: Prioritizing non-coding variants in rare disease diagnosis and interpreting variants of uncertain significance (VUS) in clinical reports.

Generates high-fidelity artificial DNA sequences that preserve the statistical properties of natural genomes. By sampling from the learned conditional probability distribution, researchers can create synthetic promoters, enhancers, or even entire gene bodies.

• Use Case: Designing libraries of synthetic regulatory elements with optimized expression levels for metabolic engineering.
• Benefit: Bypasses the limitations of natural sequence diversity and enables the creation of privacy-preserving genomic datasets for algorithm development.

Uses perplexity scoring to identify evolutionarily constrained genomic regions. A low perplexity score indicates the sequence conforms to the model's learned grammar, suggesting strong purifying selection.

• Analysis: Scanning a genome with a sliding window and flagging regions where the model exhibits high surprise (high perplexity) can reveal novel functional elements.
• Advantage: This approach is fully reference-free and does not require multi-species alignments, making it applicable to non-model organisms.

Systematically computes the predicted impact of every possible single-nucleotide substitution at a given locus. An autoregressive model efficiently scores all three alternate alleles at each position by measuring the shift in sequence log-likelihood.

• Output: A high-resolution map of nucleotide-level constraint that highlights critical bases within transcription factor binding sites or splice junctions.
• Efficiency: This computational approach accelerates saturation mutagenesis scoring by orders of magnitude compared to experimental deep mutational scanning.

Fills in missing nucleotides in incomplete genomic assemblies by autoregressively generating the most probable continuation of a sequence. The model conditions on the upstream context to predict the next base, extending contigs across ambiguous or gapped regions.

• Application: Improving the quality of draft genomes assembled from low-coverage sequencing data.
• Technique: Uses sequence corruption strategies during fine-tuning to make the model robust to the types of errors found in raw sequencing reads.

Applies a model pretrained on a data-rich species (e.g., human or mouse) to analyze a data-poor organism. The autoregressive grammar learned from the source genome captures universal biological principles that transfer across evolutionary distances.

• Validation: Fine-tuning the pretrained model on a small amount of target species data yields superior performance compared to training from scratch.
• Impact: Democratizes advanced genomic analysis for agricultural and conservation genomics where large training corpora are unavailable.

The probability assigned to a genomic sequence by an autoregressive model, computed as the product of conditional probabilities for each nucleotide given its preceding context. This metric serves as a measure of how well a sequence conforms to learned patterns of natural DNA. Low log-likelihood indicates evolutionary constraint or potential pathogenicity, making it a powerful zero-shot variant effect predictor.

A metric derived from the model's cross-entropy loss that quantifies how surprised the model is by a given sequence. In genomics, lower perplexity indicates higher confidence in the sequence's conformity to learned regulatory grammar. Perplexity is widely used to identify functional elements and measure evolutionary constraint without task-specific fine-tuning.

A computational technique that systematically introduces virtual mutations at every position in a DNA sequence and measures the resulting change in model predictions. For autoregressive models, this involves computing the shift in log-likelihood for each possible nucleotide substitution. The resulting saturation mutagenesis map identifies nucleotides critical for regulatory function and reveals binding site motifs.

An adaptive tokenization algorithm that builds a subword vocabulary by iteratively merging the most frequent pairs of nucleotides. Unlike fixed k-mer tokenization, BPE learns variable-length tokens that capture common genomic motifs like TATA boxes or splice sites as single tokens. This enables autoregressive models to handle open-vocabulary sequences and reduces sequence length for more efficient next-token prediction.