ESM-2

ESM-2 scales up to 15 billion parameters (ESM-2 15B), making it one of the largest protein language models available. It uses a standard transformer encoder architecture with rotary position embeddings and FlashAttention for efficient training.

• ESM-2 8M: 8 million parameters, 320 hidden dimensions, 6 layers
• ESM-2 35M: 35 million parameters, 480 hidden dimensions, 12 layers
• ESM-2 150M: 150 million parameters, 640 hidden dimensions, 30 layers
• ESM-2 650M: 650 million parameters, 1280 hidden dimensions, 33 layers
• ESM-2 3B: 3 billion parameters, 2560 hidden dimensions, 36 layers
• ESM-2 15B: 15 billion parameters, 5120 hidden dimensions, 48 layers

ESM-2 is pre-trained on UniRef50, a clustered version of the UniProt database that groups sequences at 50% sequence identity to reduce redundancy while preserving evolutionary diversity. This dataset contains over 65 million protein sequences spanning all domains of life.

• Eliminates over-representation of highly similar sequences
• Preserves diverse evolutionary signals across protein families
• Enables the model to learn generalizable representations rather than memorizing near-duplicates

ESM-2 uses the masked language modeling (MLM) pre-training objective, where random amino acid residues in a sequence are masked and the model must predict the original residue from the surrounding context.

• Masking rate: 15% of residues are randomly selected
• Replacement strategy: 80% masked token, 10% random amino acid, 10% unchanged
• Forces the model to learn bidirectional context and residue-residue dependencies
• Enables zero-shot variant effect prediction by comparing likelihoods of wild-type vs. mutant sequences

A landmark finding of ESM-2 is that protein structure prediction emerges naturally as model scale increases. The 15B parameter model achieves atomic-resolution structure prediction using only a linear projection on top of internal representations, without any explicit structural supervision.

• Internal attention maps correspond to contact maps of the folded protein
• The model learns to represent 3D structure implicitly through sequence patterns alone
• Enables single-sequence structure prediction without requiring multiple sequence alignments (MSAs)
• Achieves competitive accuracy with MSA-dependent methods like AlphaFold2 at significantly faster inference speeds

ESM-2 can predict the functional impact of amino acid mutations without any task-specific training. By scoring the log-likelihood ratio between the wild-type and mutant sequences, the model identifies deleterious mutations.

• Scoring formula: log P(mutant) - log P(wild-type) summed across the mutated position and its context window
• Correlates strongly with experimental deep mutational scan data across dozens of proteins
• Enables rapid in silico mutagenesis for protein engineering and clinical variant interpretation
• Particularly effective for identifying loss-of-function and pathogenic variants

ESM-2 produces per-residue embeddings that capture structural, functional, and evolutionary properties of each amino acid position. These embeddings serve as powerful features for downstream prediction tasks.

• Residue-level embeddings: Vector representations for each position in the input sequence
• Sequence-level embeddings: Aggregated representations (mean pooling or CLS token) for whole-protein tasks
• Applications include:
  • Secondary structure prediction (Q3 accuracy > 85%)
  • Subcellular localization prediction
  • Enzyme Commission number classification
  • Gene Ontology term annotation
  • Protein-protein interaction prediction

ESM-2 excels at predicting the functional impact of amino acid substitutions without any fine-tuning. By comparing the log-likelihood of the wild-type sequence against the mutated sequence using the masked language modeling objective, the model scores the deleteriousness of a variant. This approach rivals supervised methods like Deep Mutational Scan (DMS) correlations, achieving state-of-the-art performance on clinical variant databases such as ClinVar. The method is particularly powerful for rare disease diagnosis and interpreting variants of uncertain significance (VUS) in human genetics.

ESM-2 embeddings encode functional properties that enable accurate prediction of Enzyme Commission (EC) numbers and Gene Ontology (GO) terms. By training lightweight linear probes or shallow classifiers on top of frozen ESM-2 representations, researchers can annotate uncharacterized proteins in newly sequenced genomes. This is critical for biodiscovery pipelines searching for novel PETases, cellulases, or other industrially relevant biocatalysts in environmental metagenomes.

The contextual embeddings from ESM-2 capture signal peptides, transmembrane helices, and other localization motifs. A simple classifier trained on these representations can determine whether a protein is destined for the mitochondria, endoplasmic reticulum, nucleus, or extracellular space. This application is essential for drug target identification, as the therapeutic tractability of a target depends heavily on its cellular compartment.

ESM-2 embeddings can be used to predict developability attributes critical for biologic drug manufacturing and industrial enzyme design. Models trained on experimental solubility and thermostability data using ESM-2 representations guide the engineering of proteins with improved expression yields and resistance to aggregation. This reduces the experimental burden in directed evolution campaigns by prioritizing variants with favorable biophysical profiles.

By navigating the latent space of ESM-2, researchers perform semantic mutagenesis—generating novel protein sequences with altered properties by interpolating between or perturbing learned representations. This technique allows the exploration of the fitness landscape beyond natural sequences, generating functional variants that retain structural integrity while introducing desired modifications. It is a foundational method for de novo protein design and functional diversification.