Evolutionary Scale Modeling (ESM-2)

ESM-2 is trained using a masked language modeling (MLM) objective on millions of evolutionarily diverse protein sequences. During training, random amino acid residues are masked, and the model learns to predict the original residue from the surrounding sequence context. This forces the model to internalize the biochemical grammar of proteins—including residue co-evolution patterns, secondary structure propensities, and functional site constraints—without any explicit structural supervision. The resulting representations capture deep evolutionary information that generalizes across diverse downstream tasks.

ESM-2 replaces traditional absolute position encodings with Rotary Position Embeddings (RoPE), which encode positional information by rotating the query and key vectors in the self-attention mechanism. This approach naturally captures the relative distance between amino acid residues rather than their absolute positions, enabling the model to generalize to sequence lengths unseen during training. RoPE is particularly well-suited for protein sequences, where the functional relationship between residues depends on their spatial proximity in the folded structure, not their linear separation.

A defining discovery of the ESM family is that protein structure prediction emerges as a property of scale. As ESM-2 scales from 8 million to 15 billion parameters, the attention heads spontaneously learn to represent residue-residue contacts in three-dimensional space. The 15B parameter ESM-2 achieves atomic-resolution structure prediction directly from its internal representations using a lightweight folding head, without requiring multiple sequence alignments (MSAs) or template-based inputs. This enables single-sequence structure prediction at speeds orders of magnitude faster than traditional methods.

ESM-2 can predict the functional impact of amino acid mutations without any task-specific fine-tuning. By computing the log-likelihood ratio between the wild-type and mutated sequences under the model's learned distribution, ESM-2 produces a zero-shot mutation score that correlates strongly with experimental measurements of protein stability and pathogenicity. This capability enables:

• Variant prioritization for clinical genomics
• Stability engineering for industrial enzymes
• Pathogenicity screening for rare disease variants
• Deep mutational scanning in silico

ESM-2 is trained on the UniRef50 dataset, a clustered version of the UniProt knowledgebase that groups sequences at 50% sequence identity. This clustering reduces redundancy while preserving the vast diversity of the protein universe, exposing the model to approximately 65 million unique protein sequences spanning all domains of life. The use of UniRef50 ensures that the model learns generalizable features rather than memorizing near-duplicate sequences, improving its ability to generalize to novel proteins and distant homologs.

The self-attention mechanism in ESM-2 provides a built-in interpretability tool for biological discovery. Attention heatmaps reveal which residues the model considers most relevant for a given prediction, often corresponding to:

• Catalytic sites in enzyme active sites
• Binding interfaces in protein-protein interactions
• Allosteric networks that transmit conformational signals
• Evolutionarily coupled residues that co-vary across species These attention patterns can guide mutagenesis experiments and reveal functional mechanisms without wet-lab screening.

A transformer model trained on massive databases of protein amino acid sequences using self-supervised objectives. It learns the underlying grammar of protein structure and function to generate informative embeddings. ESM-2 is a specific, highly performant instance of a pLM. These models treat amino acid sequences analogously to natural language text, capturing long-range dependencies and residue-residue interactions that dictate folding.

The core self-supervised pre-training objective used by ESM-2. A random subset of amino acids in a protein sequence is masked, and the model learns to predict the original residues from the surrounding evolutionary context. This forces the model to learn fundamental sequence conservation patterns, biochemical constraints, and structural couplings without requiring any labeled structural data.

A key capability of ESM-2 where the model predicts the effect of a mutation using only the difference in sequence likelihood (log-odds ratio) between the wild-type and mutant sequences. No supervised fine-tuning on labeled variant effect data is required. This enables:

• Variant effect scoring for clinical genomics
• In-silico deep mutational scanning
• Identification of pathogenic mutations in proteins of unknown structure

The task of determining which pairs of amino acid residues are in close spatial proximity within a folded protein's 3D structure. This capability emerges in the attention heads of ESM-2 during pre-training. The resulting contact maps are foundational for de novo structure prediction and are derived directly from the co-evolutionary signal captured in the model's self-attention weights.

The positional encoding method used in ESM-2's transformer architecture. RoPE encodes position by rotating the query and key vectors, naturally capturing the relative distance between amino acid residues. This allows ESM-2 to extrapolate to longer protein sequences than those seen during training and effectively model long-range interactions between distal residues.

A measure of the degree to which an amino acid position remains unchanged across evolutionary time. ESM-2 learns this fundamental signal during self-supervised pre-training on millions of diverse sequences. Positions with high conservation scores in the model's representations correlate strongly with functional importance, such as active sites, binding interfaces, and structurally critical core residues.