Protein Language Model

The core pretraining task randomly masks a subset of amino acids and trains the model to predict the original residues from surrounding sequence context.

• Bidirectional context captures dependencies from both N-terminal and C-terminal directions simultaneously
• Residue-level supervision forces the model to learn local physicochemical constraints and long-range co-evolutionary couplings
• Corruption strategies include random mutation, deletion, and insertion of residues to improve robustness

This objective mirrors BERT-style pretraining in natural language processing, treating amino acid sequences as the fundamental tokens of the protein grammar.

Transformer attention heads implicitly learn to identify residue pairs in spatial proximity by analyzing co-evolutionary signals embedded in the input multiple sequence alignment.

• Attention patterns correlate with true inter-residue distances, enabling zero-shot contact map extraction
• Symmetrization of raw attention matrices improves contact prediction accuracy by enforcing reciprocal relationships
• Long-range contacts spanning more than 24 sequence positions are captured through deeper transformer layers

The linear relationship between attention head activations and 3D proximity provides an interpretable window into how the model internalizes structural information without explicit geometric supervision.

Protein language models quantify the functional impact of amino acid substitutions by computing the log-likelihood ratio between wild-type and mutant residues conditioned on the surrounding sequence context.

• Scoring protocol masks the mutated position and compares predicted probabilities for reference vs. alternative amino acids
• Spearman correlation with deep mutational scanning experiments reaches 0.5–0.7 across diverse protein families
• Clinical variant interpretation leverages these scores to prioritize pathogenic missense mutations in human disease genes

The zero-shot capability eliminates the need for variant-specific training data, enabling generalization to proteins without prior functional characterization.

Residue-level embeddings encode biochemical properties and functional annotations that can be transferred across homologous sequences through representation similarity.

• Secondary structure prediction from frozen embeddings achieves accuracy competitive with dedicated tools without fine-tuning
• Binding site identification emerges in embedding space as clusters of residues with distinct physicochemical signatures
• Phosphorylation and PTM sites are detectable through subtle patterns in local sequence context representations
• Embedding arithmetic enables analogical reasoning: embedding(kinase_active_site) − embedding(kinase) + embedding(phosphatase) approximates phosphatase catalytic motifs

The dense vector representations serve as a unified feature space for downstream prediction tasks, reducing the need for task-specific feature engineering.

Aggregated attention patterns across transformer layers reveal residues that participate in intermolecular interfaces, enabling binding site prediction without structural data.

• Column-wise attention in MSA-based models identifies positions under strong evolutionary constraint from interaction partners
• Entropy analysis of attention distributions distinguishes interface residues from solvent-exposed non-functional positions
• Cross-protein attention in paired-sequence architectures directly models residue-residue couplings across binding partners
• Conservation-aware normalization corrects for background sequence conservation to isolate interaction-specific signals

This approach enables computational identification of druggable pockets and protein-protein interaction interfaces from sequence alone, accelerating target validation in early-stage drug discovery.

Autoregressive decoding from protein language models generates novel amino acid sequences conditioned on desired functional constraints or structural templates.

• Template-conditioned generation produces sequences predicted to fold into specified backbone structures by iteratively sampling residues
• Functional motif scaffolding generates full protein sequences that embed specified catalytic or binding motifs within stable folds
• Temperature-controlled sampling modulates sequence diversity, balancing exploration of novel functional variants against preservation of structural integrity
• Perplexity filtering removes generated sequences with low model confidence, enriching for designs likely to express and fold correctly

The generative capability transforms protein language models from analytical tools into de novo design engines, enabling creation of enzymes, binders, and therapeutics with tailored properties.

A family of large-scale transformer models trained on 250 million protein sequences using masked language modeling. The 15B-parameter variant learns deeply contextualized residue representations that capture secondary structure, binding sites, and remote homology without requiring multiple sequence alignments.

• Key innovation: Attention patterns spontaneously recapitulate the structure of the protein folding landscape.
• Scale: Scales to 15B parameters, enabling metagenomic structure prediction via ESMFold.

A suite of autoregressive protein language models trained on over 1 billion protein sequences from genomic, metagenomic, and synthetic sources. ProGen2 generates novel, functional protein sequences conditioned on controlled tags specifying properties like organism, cellular compartment, and enzymatic function.

• Key innovation: Conditional generation enables fine-grained control over sequence properties.
• Scale: Ranges from 151M to 6.4B parameters, demonstrating predictable scaling laws for protein generation.

An autoregressive transformer trained on ~50 million non-redundant protein sequences that generates novel, globular protein sequences with amino acid compositions and secondary structure propensities statistically indistinguishable from natural proteins.

• Key innovation: Generated sequences are predicted by AlphaFold2 to fold into stable, well-packed structures with high pLDDT scores.
• Application: Exploration of uncharted regions of protein sequence space for de novo enzyme design.

A bidirectional transformer that jointly learns from protein sequences and associated Gene Ontology (GO) annotations using a hybrid masked language modeling and annotation prediction objective. This dual-task training produces representations that excel at function prediction and remote homology detection.

• Key innovation: Integrates structured biological annotations directly into the pre-training signal.
• Architecture: Combines local (sequence) and global (annotation) attention mechanisms for holistic protein understanding.

A protein language model specifically designed for variant effect prediction that combines a large autoregressive transformer with a separate retrieval-based module for scoring rare mutations. The tiered architecture enables state-of-the-art zero-shot fitness prediction across deep mutational scanning benchmarks.

• Key innovation: Retrieval-augmented scoring for mutations poorly represented in training data.
• Performance: Achieves top ranks in the ProteinGym benchmark for variant effect prediction.

A protein language model that leverages optimal transport theory to align representations across different model scales, enabling efficient knowledge transfer from large teacher models to smaller student architectures. Ankh achieves competitive performance on structure and function prediction tasks with substantially reduced computational requirements.

• Key innovation: Cross-scale representation alignment via sliced Wasserstein distance.
• Efficiency: Delivers strong performance on remote homology detection and secondary structure prediction at reduced inference cost.