ProtGPT2

ProtGPT2 generates protein sequences residue-by-residue, conditioning each amino acid prediction on all previously generated tokens. This left-to-right decoding captures long-range dependencies and produces sequences that fold into globular, stable structures.

• Trained on ~50 million non-redundant UniRef50 sequences
• Uses byte pair encoding (BPE) for an efficient subword vocabulary
• Generates sequences with perplexity scores comparable to natural proteins

Unlike traditional methods requiring multiple sequence alignments (MSAs), ProtGPT2 learns evolutionary constraints directly from raw sequence data. Generated sequences exhibit:

• Predicted Local Distance Difference Test (pLDDT) scores indicating well-folded structures
• Secondary structure distributions matching natural proteins
• Contact maps consistent with globular topologies

This MSA-free approach dramatically accelerates design cycles.

ProtGPT2's learned protein embedding space enables directed navigation for functional design. Semantic mutagenesis allows researchers to:

• Interpolate between protein families to discover chimeric sequences
• Perturb latent representations to modulate properties like thermostability
• Sample from high-likelihood regions for fitness landscape exploration

The model captures a continuous manifold of protein sequence space.

While natively unconditional, ProtGPT2's architecture supports control tag conditioning—prepending functional tokens to guide generation toward specific protein families or properties.

• Prefix with Pfam domain identifiers for family-specific generation
• Condition on Gene Ontology (GO) terms for function-guided design
• Extendable to property tags like EC number or subcellular localization

This enables targeted library design for enzyme engineering and therapeutic discovery.

ProtGPT2-generated sequences match or exceed natural proteins across key quality metrics:

• Sequence recovery rate comparable to native validation sets
• Generated sequences pass AlphaFold2 structure prediction with high confidence
• TM-score distributions indicate novel folds not memorized from training data
• Outperforms variational autoencoders in sequence diversity and foldability

Unlike ProtBERT or ESM-2 which use masked language modeling (MLM) for representation learning, ProtGPT2's autoregressive objective is optimized for generation.

• MLM models excel at variant effect prediction and contact prediction
• ProtGPT2 excels at producing novel, full-length sequences
• Complementary to ProteinMPNN which solves inverse folding given a backbone

Use ProtGPT2 for de novo design; use MLM models for analysis and scoring.

The foundational generative mechanism used by ProtGPT2, where protein sequences are produced one amino acid at a time. Each residue prediction is conditioned on all previously generated tokens, enabling the model to capture long-range dependencies. This contrasts with masked language modeling (used by ESM-2 and ProtBERT), which is bidirectional. Autoregressive models are naturally suited for de novo design because they can generate sequences from scratch without requiring a partially complete scaffold.

A family of large-scale autoregressive models closely related to ProtGPT2, trained on over one billion sequences. ProGen2 introduces conditioning tags that allow users to specify desired protein families or functional properties during generation. While ProtGPT2 focuses on generating structurally plausible sequences, ProGen2 emphasizes controllable generation across diverse evolutionary lineages. Both models demonstrate that scaling autoregressive transformers on protein data yields fluent, novel sequences.

A critical quality metric for evaluating ProtGPT2 outputs. Perplexity quantifies how surprising a generated sequence is under the model's learned distribution. Low perplexity indicates the sequence conforms to the grammatical rules of natural proteins. This metric is also used for zero-shot variant effect prediction: mutations that increase perplexity relative to the wild-type are likely deleterious. ProtGPT2's low perplexity on generated sequences confirms its ability to produce realistic proteins.

A technique for navigating ProtGPT2's latent space to design proteins with altered properties. By interpolating between learned representations or applying targeted perturbations, researchers can generate sequences that maintain structural plausibility while exploring new functional regions. This method leverages the model's internal understanding of sequence-structure relationships to perform directed evolution in silico, bypassing the need for extensive laboratory screening.

A conceptual mapping of all possible protein sequences to their biological function. ProtGPT2 learns a smoothed approximation of natural fitness landscapes from evolutionary data. Generated sequences occupy regions of high probability under this learned landscape, increasing the likelihood of experimental success. Understanding fitness landscapes helps researchers visualize how ProtGPT2's generations navigate the vast sequence space while avoiding non-functional regions.

While ProtGPT2 generates sequences unconditionally, ProteinMPNN solves the inverse folding problem: predicting an amino acid sequence given a target backbone structure. These approaches are complementary. ProtGPT2 excels at exploring novel folds and sequences, while ProteinMPNN provides robust sequence design for fixed structural templates. Together, they form a powerful pipeline for structure-aware protein engineering.