Evolutionary Scale Modeling (ESM)

ESM models are pretrained using a masked language modeling (MLM) objective on large, diverse protein sequence databases such as UniRef. During training, random amino acids are masked, and the model learns to predict the original residue from the surrounding sequence context. This forces the network to internalize the complex grammar of protein sequences, including evolutionary covariation, biochemical constraints, and structural propensities. Unlike traditional methods that require multiple sequence alignments (MSAs) as input, ESM captures this co-evolutionary information within its learned parameters, enabling single-sequence structure prediction.

A defining capability of ESM models is the ability to predict three-dimensional protein structure directly from a single amino acid sequence, without requiring a multiple sequence alignment (MSA). The model's attention heads have been shown to represent contacts between residues that are proximal in the folded 3D structure. By interpreting these attention patterns as a proxy for residue-residue distance constraints, ESM-fold can generate accurate structural predictions. This single-sequence approach dramatically accelerates inference, making it feasible to predict structures for metagenomic proteins and other sequences for which deep MSAs cannot be constructed.

ESM models can assess the functional impact of amino acid substitutions without any task-specific fine-tuning. The zero-shot variant effect score is computed as the log-likelihood ratio between the reference (wild-type) and alternate (mutant) amino acids at a given position, conditioned on the surrounding sequence context. This approach leverages the model's learned distribution over natural protein sequences: mutations that are highly probable under the model are considered benign, while those that are improbable are predicted to be deleterious. This capability enables high-throughput in silico mutagenesis and clinical variant interpretation.

The self-attention mechanisms within ESM's Transformer layers learn to attend strongly to residue pairs that are in spatial contact within the folded protein structure. By extracting and interpreting these attention patterns, the model generates a contact map that serves as a foundation for structure prediction. Key features include:

• Symmetrization of raw attention matrices to enforce reciprocal contact predictions
• Regression from multi-head attention to a single probability of contact
• Long-range contact capture between residues separated by hundreds of positions in the linear sequence This learned representation of evolutionary covariation is what enables single-sequence structure prediction.

ESM models produce contextualized residue-level embeddings that encode biochemical properties, structural context, and evolutionary information into dense vector representations. These embeddings serve as transferable features for a wide range of downstream tasks, including:

• Protein function prediction (Gene Ontology term classification)
• Subcellular localization prediction
• Protein-protein interaction site identification
• Thermostability and biophysical property prediction The embeddings capture information that generalizes across protein families, enabling strong performance even on understudied proteins with limited experimental annotations.

The single-sequence inference speed of ESM-fold enables structural characterization of proteins at an unprecedented scale. This has been applied to metagenomic datasets—genetic material recovered directly from environmental samples—to predict structures for hundreds of millions of previously uncharacterized proteins. This capability reveals the structural diversity of the microbial dark proteome, uncovering novel folds and functional sites that are absent from curated databases. The resulting structural atlas provides a resource for enzyme discovery, drug target identification, and understanding the functional potential of microbial communities.