ProGen2

ProGen2 employs a transformer-based autoregressive decoding architecture trained on over 1 billion protein sequences from diverse genomic, metagenomic, and immune repertoire databases. The model learns to predict the next amino acid given all preceding residues, capturing long-range dependencies and evolutionary constraints. This scale enables the model to internalize a comprehensive fitness landscape spanning all known protein families, resulting in state-of-the-art perplexity scores that correlate strongly with experimental stability and function.

A defining innovation of ProGen2 is the use of conditioning tags prepended to input sequences during training and generation. These tags encode metadata such as:

• Taxonomic lineage (e.g., [Bacteria], [Eukaryota], [Mammalia])
• Cellular compartment (e.g., [Cytoplasm], [Membrane])
• Functional annotations (e.g., [Enzyme], [Transcription Factor]) This allows users to steer generation toward proteins with specific organismal origins or functional properties, enabling controlled semantic mutagenesis across the latent space.

ProGen2 can perform zero-shot variant effect prediction by computing the log-likelihood ratio between wild-type and mutated sequences. Without any task-specific fine-tuning, the model scores the functional impact of single amino acid substitutions, achieving strong correlation with deep mutational scan experimental data. This capability enables rapid in silico screening of mutations for protein engineering, identifying stabilizing or destabilizing variants before committing to costly wet-lab experiments.

The ProGen2 family includes models at multiple scales to balance performance and computational cost:

• ProGen2-small (151M parameters): Fast inference for high-throughput screening
• ProGen2-medium (764M parameters): Balanced performance for most design tasks
• ProGen2-large (2.7B parameters): High-fidelity generation and variant scoring
• ProGen2-xlarge (6.4B parameters): Maximum representational capacity for challenging design problems Each model shares the same Byte Pair Encoding tokenizer optimized for amino acid sequences, enabling seamless scaling across the family.

ProGen2 supports both unconditional generation—producing novel protein sequences that resemble the natural distribution—and conditional generation guided by taxonomic or functional tags. Generated sequences exhibit:

• High sequence recovery rates when benchmarked against inverse folding models
• Structural plausibility validated by AlphaFold2 and ESMFold predictions
• Diversity comparable to natural protein families This dual-mode generation makes ProGen2 suitable for both exploratory design and targeted engineering of enzymes, antibodies, and therapeutic proteins.

Beyond generation, ProGen2 produces rich protein embeddings from its final hidden layers that capture structural, functional, and evolutionary information. These fixed-length vector representations can be extracted and used as input features for supervised predictors of:

• Enzyme Commission number classification
• Gene Ontology term annotation
• Subcellular localization prediction
• Thermostability and solubility regression The embeddings transfer effectively across diverse protein engineering tasks without requiring task-specific architecture modifications.

The generative mechanism underlying ProGen2, where sequences are produced one amino acid at a time. Each residue prediction is conditioned on all previously generated tokens in the chain.

• Left-to-right generation: Models causal dependencies in the primary sequence.
• Contrast with MLMs: Unlike masked models (e.g., ESM-2), autoregressive models are optimized for de novo sequence generation rather than representation learning.
• Conditioning tags: ProGen2 uses special control tags to steer the autoregressive process toward specific protein families or functions.

A subword tokenization algorithm adapted for amino acid sequences that segments proteins into frequent multi-residue tokens. This enables more efficient vocabulary representation in large language models like ProGen2.

• Reduces sequence length compared to single-residue tokenization.
• Captures common motifs (e.g., 'EL', 'LK', 'AAA') as single tokens.
• Improves computational efficiency during training and inference.
• ProGen2's massive scale (1B+ sequences) benefits significantly from this compressed representation.

A metric derived from language models that quantifies how surprising or unlikely a given amino acid sequence is under the model's learned distribution. ProGen2 uses perplexity to assess sequence quality and variant effects.

• Low perplexity: Sequence is highly probable under the model's learned protein grammar.
• Variant effect prediction: Mutations that increase perplexity are likely destabilizing or deleterious.
• Zero-shot capability: No task-specific training data required; the model's pre-trained distribution serves as a proxy for evolutionary fitness.

A conceptual mapping of all possible protein sequences to their associated biological fitness or functional activity. ProGen2 learns a smoothed approximation of this landscape from evolutionary data.

• Sequence space: The vast combinatorial space of possible amino acid sequences.
• Peaks and valleys: High-fitness sequences cluster in regions ProGen2 learns to sample from.
• Guided generation: Conditioning tags allow navigation toward specific functional peaks.
• ProGen2's autoregressive sampling effectively performs a biased random walk across this learned landscape.

The process of navigating a protein language model's latent space to generate novel sequences with altered properties. With ProGen2, this involves interpolating or perturbing learned representations.

• Latent space interpolation: Blending embeddings of two functional proteins to generate chimeric sequences.
• Tag-based steering: Modifying conditioning tags to shift generated sequences toward new functional families.
• Directed evolution in silico: Semantic mutagenesis enables exploration of sequence diversity without laborious experimental screening.
• ProGen2's large-scale training provides a rich, continuous latent space for these operations.

The percentage of native amino acid residues correctly predicted by an inverse folding model, serving as a standard benchmark metric for protein design accuracy.

• Inverse folding context: Given a backbone structure, how well does the model recover the native sequence?
• ProGen2's role: While primarily a forward-generation model, ProGen2's learned representations can be fine-tuned for structure-conditioned design tasks.
• Benchmark comparison: Models like ProteinMPNN specialize in this metric, but ProGen2's generative diversity complements structure-focused approaches.
• High recovery rates indicate the model has internalized the sequence-structure relationship.