Protein Language Model

PLMs are trained on hundreds of millions of protein sequences using objectives that require no manual labels. Masked language modeling randomly hides amino acids and trains the model to predict them from surrounding context, forcing it to learn co-evolutionary couplings, secondary structure propensities, and residue contact patterns. Autoregressive generation predicts each residue sequentially, enabling de novo sequence design. Training corpora typically span UniRef clusters, Pfam domains, and metagenomic databases to capture the full diversity of natural protein space.

Each amino acid position in a sequence receives a dense vector representation that encodes its biochemical context. These per-residue embeddings capture:

• Local structural propensities (helix, sheet, loop)
• Evolutionary conservation signals
• Long-range contact relationships
• Functional site membership

Embeddings from final transformer layers serve as universal representations transferable to downstream tasks including variant effect prediction, subcellular localization, and enzyme function classification without task-specific fine-tuning.

Transformer attention heads naturally learn to attend to structurally and functionally coupled residue pairs. Attention map analysis reveals that specific heads specialize in:

• Short-range local structure contacts
• Long-range tertiary contacts across the folded protein
• Binding interface residues

This emergent property enables contact prediction without explicit structural supervision. Extracted attention patterns achieve accuracy competitive with dedicated co-evolutionary methods like direct coupling analysis, demonstrating that sequence alone contains sufficient information to infer 3D proximity.

Autoregressive PLMs like ProtGPT2 and ProGen2 generate novel protein sequences by sampling from learned sequence distributions. These models produce sequences that:

• Fold into stable globular structures confirmed by AlphaFold
• Maintain conserved functional motifs
• Explore sequence space beyond natural diversity

Conditioning tags enable controlled generation of specific protein families, enzyme classes, or desired properties. Generated sequences show experimental expression and solubility rates comparable to natural proteins, validating the models' capture of fundamental folding constraints.

PLMs evaluate mutational consequences by comparing the pseudo-log-likelihood of wild-type and mutant sequences. The scoring principle: if a mutation makes the sequence less probable under the model's learned distribution, it likely disrupts structure or function. This zero-shot approach:

• Requires no experimental variant effect data
• Correlates with deep mutational scan measurements
• Generalizes across proteins and mutation types

Performance rivals supervised methods trained on thousands of assayed variants, making PLMs powerful tools for clinical variant interpretation and protein engineering prioritization.

Unlike natural language, protein sequences have a small alphabet of 20 standard amino acids. Tokenization approaches include:

• Character-level: Each residue is a token; simple but limits context window efficiency
• Byte Pair Encoding (BPE): Merges frequent residue pairs into multi-residue tokens, capturing common motifs like catalytic triads or structural patterns
• Physicochemical grouping: Tokens based on residue properties (hydrophobic, charged, polar)

BPE tokenization reduces sequence length by 30-50%, enabling longer effective context windows while preserving the model's ability to attend to individual residues when needed.

Meta AI's flagship protein language model family trained with masked language modeling on hundreds of millions of sequences. ESM-2 scales to 15 billion parameters, learning deep evolutionary patterns that enable atomic-resolution structure prediction directly from sequence. Its embeddings capture biophysical properties and long-range contacts without requiring multiple sequence alignments, making it a workhorse for zero-shot variant effect prediction and enzyme function annotation.

A BERT-based model pre-trained on UniRef100 using masked language modeling to produce residue-level embeddings that capture local and global sequence context. ProtBERT excels at secondary structure prediction, subcellular localization, and contact prediction when fine-tuned. Its bidirectional attention mechanism allows each amino acid representation to be informed by the entire surrounding sequence, making it particularly effective for tasks requiring holistic protein understanding.

An autoregressive decoder based on GPT-2 that generates novel protein sequences by sampling from the learned distribution of natural proteins. ProtGPT2 produces sequences with globular fold topologies and experimentally validated stability. Key capabilities include:

• Generation of sequences with low sequence identity to natural proteins
• Preservation of structural plausibility and folding potential
• Exploration of uncharted regions of protein fitness landscapes

A suite of autoregressive models trained on over one billion protein sequences with taxonomic and functional conditioning tags. ProGen2 enables controlled generation of proteins from specific families, organisms, or functional classes. Its conditioning mechanism allows users to specify desired properties—such as enzyme class or organism source—and generate sequences that satisfy those constraints while maintaining structural viability.

A message-passing neural network that solves the inverse protein folding problem: given a target backbone structure, predict the amino acid sequence that will fold into it. ProteinMPNN achieves exceptional sequence recovery rates and generates sequences that experimentally fold to the desired structure with high success. Its architecture encodes spatial relationships between residues as a graph, enabling robust design even for challenging protein topologies.

A protein language model that achieves competitive performance with significantly fewer parameters by leveraging attention-based downsampling and efficient pre-training strategies. Ankh demonstrates that careful architectural design can match or exceed larger models on tasks including fluorescence prediction, stability prediction, and remote homology detection, making it suitable for resource-constrained deployment environments.

Meta AI's Evolutionary Scale Modeling transformer that uses masked language modeling on millions of protein sequences to generate embeddings and predict structure, function, and variant effects with state-of-the-art accuracy. The model scales up to 15 billion parameters and learns evolutionary patterns without requiring multiple sequence alignments.

A dense, fixed-length vector representation of a protein sequence or residue learned by a language model that captures structural, functional, and evolutionary information. These embeddings serve as transferable features for downstream tasks including:

• Variant effect prediction
• Subcellular localization
• Enzyme function classification Embeddings from models like ESM-2 and ProtBERT encode biophysical properties implicitly learned during pre-training.

The use of protein language models to score the functional impact of mutations by comparing the likelihood ratio of the wild-type sequence against the mutated sequence without any task-specific training data. This approach leverages the model's learned distribution over evolutionary sequences—mutations that reduce sequence likelihood are predicted to be deleterious. Key metric: Spearman correlation with deep mutational scan data.

The computational task of predicting an amino acid sequence that will fold into a specified three-dimensional protein backbone structure. Models like ProteinMPNN use message-passing on the structure graph to generate sequences with high recovery rates. This is the reverse of structure prediction and is fundamental to de novo protein design workflows.

A self-supervised pre-training objective where random amino acids in a sequence are masked and the model learns to predict them from the surrounding context, analogous to BERT training. This forces the model to learn:

• Local residue dependencies
• Long-range co-evolutionary couplings
• Structural constraints Used by ESM-2, ProtBERT, and most encoder-only protein language models.

A conceptual mapping of all possible protein sequences to their associated biological fitness or functional activity, used to visualize evolutionary trajectories and guide engineering. Protein language models implicitly learn a smoothed version of this landscape through exposure to evolutionary data, enabling navigation via semantic mutagenesis in latent space to discover optimized variants.