Autoregressive Protein Decoding

The model generates protein sequences one residue at a time, strictly from the N-terminus to the C-terminus. Each amino acid prediction is conditioned on all previously generated residues in the sequence. This unidirectional causal dependency mirrors natural ribosomal translation and is enforced by a triangular attention mask during training and inference. Unlike masked language models that can attend bidirectionally, the autoregressive factorization decomposes the joint probability of a sequence as:

p(x) = p(x₁) * p(x₂|x₁) * p(x₃|x₁,x₂) * ... * p(xₙ|x₁,...,xₙ₋₁)

This allows direct computation of sequence likelihoods and enables controlled generation through temperature sampling and nucleus (top-p) filtering.

The core architectural component enabling autoregressive decoding is the causal self-attention layer. During training, a lower-triangular mask is applied to the attention matrix, preventing any position from attending to future residues. This ensures the model learns to predict the next amino acid using only the prefix context. Key implementation details:

• Mask shape: Lower triangular matrix where position i can attend to positions 0 through i
• Memory complexity: O(n²) for sequence length n, managed through optimized kernels like FlashAttention
• KV caching: During inference, previously computed key-value pairs are cached to avoid redundant computation, reducing per-step latency to O(n) rather than O(n²)

This mechanism is identical in principle to GPT-style language models but operates over a vocabulary of 20 canonical amino acids plus special tokens.

Modern autoregressive protein models like ProGen2 and ProtGPT2 incorporate conditioning tags prepended to the sequence to steer generation toward specific protein families, functions, or taxonomic origins. These tags are treated as part of the prefix context and influence all subsequent residue predictions. Common conditioning modalities include:

• Pfam domain identifiers: Direct generation toward specific protein families
• Gene Ontology terms: Control molecular function or biological process
• Taxonomic lineage tags: Bias generation toward sequences from specific organisms
• Numerical property tags: Condition on values like thermostability or solubility

During inference, the tag sequence is provided as a fixed prefix, and the model generates the protein sequence autoregressively conditioned on that prefix. This enables zero-shot generation of proteins with desired characteristics without retraining.

A direct consequence of the autoregressive factorization is the ability to compute exact sequence perplexity—a measure of how well the model's learned distribution explains a given protein. Perplexity is calculated as:

PPL = exp(-1/N * Σ log p(xᵢ | x_{<i}))

Lower perplexity indicates the sequence is more 'natural' under the model's learned distribution. This metric enables several critical applications:

• Variant effect prediction: Compare wild-type and mutant perplexities to score mutational impact without any supervised fine-tuning
• Sequence quality filtering: Reject generated sequences with anomalously high perplexity
• Domain boundary detection: Identify regions where perplexity sharply changes, indicating domain transitions
• Orphan sequence assessment: Evaluate whether novel sequences belong to known protein families

This zero-shot capability is a key advantage over masked language models, which require iterative decoding to estimate likelihoods.

The stochasticity and diversity of generated sequences are controlled through two primary sampling parameters applied to the output logits at each decoding step:

• Temperature (τ): Scales logits before softmax. Lower values (τ < 1.0) sharpen the distribution, producing more conservative, high-confidence sequences. Higher values (τ > 1.0) flatten the distribution, increasing diversity but risking structural implausibility. Typical protein generation uses τ ∈ [0.7, 1.2]
• Top-p (Nucleus) Sampling: Only considers the smallest set of tokens whose cumulative probability exceeds threshold p. This dynamically truncates the low-probability tail, preventing sampling from the unreliable long tail of the distribution. Common thresholds: p ∈ [0.9, 0.95]

These parameters allow practitioners to navigate the trade-off between sequence novelty and structural plausibility, analogous to controlling creativity in text generation.

The model is trained using a standard causal language modeling objective—minimizing the cross-entropy loss between predicted and actual next residues across all positions. For a sequence of length N, the loss is:

L = -1/N * Σ log p(xᵢ | x_{<i}; θ)

Key training considerations:

• Dataset scale: Models like ProGen2 are trained on over 1 billion protein sequences from curated databases including UniRef, Pfam, and metagenomic sources
• Sequence length handling: Proteins range from ~50 to >1000 residues; models use learned positional encodings or rotary position embeddings to handle variable lengths
• Special tokens: Sequences are delimited with start-of-sequence and end-of-sequence tokens; the model learns to generate the EOS token to terminate decoding
• Data augmentation: Homologous sequences are often included to improve generalization across protein families

This objective forces the model to learn the underlying grammar of protein sequences, including structural constraints, evolutionary conservation patterns, and functional motifs.

An autoregressive generative protein language model based on the GPT-2 architecture that produces novel, structurally plausible protein sequences. Trained on UniRef50, it generates sequences residue-by-residue, with each amino acid conditioned on all preceding residues. The model captures remote homology and generates proteins with stable predicted folds that are distinct from natural sequences, enabling exploration of uncharted regions of protein space.

A suite of large-scale autoregressive protein language models trained on over one billion sequences with conditioning tags. ProGen2 generates proteins across diverse families and functions by conditioning on taxonomic, functional, and structural metadata. The largest variant contains 6.4 billion parameters, making it one of the most capacious generative models for protein sequence design, capable of producing functional enzymes and structural proteins.

A metric derived from autoregressive language models that quantifies how surprising or unlikely a given amino acid sequence is under the model's learned distribution. Lower perplexity indicates higher sequence plausibility. Perplexity is used to:

• Assess variant effects by comparing mutant and wild-type likelihoods
• Filter generated sequences for quality control
• Detect anomalous domains in multi-domain proteins

The process of navigating a protein language model's latent space to generate novel sequences with altered properties. By interpolating between learned representations or applying controlled perturbations, semantic mutagenesis enables directed evolution in silico. This technique leverages the model's internal understanding of sequence-structure-function relationships to propose mutations that preserve fold stability while altering activity, specificity, or thermostability.

A subword tokenization algorithm adapted for amino acid sequences that segments proteins into frequent multi-residue tokens. Unlike single-residue tokenization, BPE captures common motifs like 'ELK' or 'GPG' as single tokens, enabling more efficient vocabulary representation. This reduces sequence length for the transformer and allows the model to learn higher-order grammatical patterns in protein sequences, improving generation quality and computational efficiency.

The percentage of native amino acid residues correctly predicted by an inverse folding or generative model, serving as a standard benchmark metric for protein design accuracy. In autoregressive decoding, recovery rate measures how well the model reconstructs natural sequences given structural or contextual constraints. Typical state-of-the-art models achieve 30-52% recovery on single-chain proteins, with higher rates indicating better capture of sequence-structure relationships.