Protein Embedding

Modern protein embeddings are contextual, meaning the vector for a specific residue changes depending on its surrounding sequence neighbors. This contrasts with older static methods like BLOSUM or one-hot encoding.

• Contextual: A cysteine in an active site gets a different vector than a cysteine in a hydrophobic core.
• Static: Every alanine has the same vector regardless of position.
• Benefit: Captures the epistatic interactions critical for variant effect prediction.

Embeddings implicitly capture evolutionary constraints without requiring explicit Multiple Sequence Alignments (MSAs). Models like ESM-2 learn co-evolutionary patterns during pre-training.

• The geometry of the embedding space reflects homologous relationships.
• Distance between two sequence embeddings correlates with their functional divergence.
• This enables zero-shot transfer to tasks like contact prediction and thermostability prediction.

Linear projections of embeddings can recover 3D structural features. The internal representations of models like ESM-2 contain enough information to predict atomic-level coordinates.

• Contact maps can be extracted via attention head analysis.
• Secondary structure (alpha-helices, beta-sheets) is linearly separable in embedding space.
• This property enables inverse folding models like ProteinMPNN to design sequences that fold into target backbones.

Embeddings exist at two granularities, each suited for different tasks.

• Residue-level embeddings: A vector per amino acid, used for binding site prediction or post-translational modification identification.
• Sequence-level embeddings: A single vector for the whole protein, obtained by averaging residue embeddings or using a special [CLS] token. Used for subcellular localization or Gene Ontology term prediction.
• Mean pooling is common but attention-weighted pooling often yields better global representations.

The embedding space is smooth and continuous, allowing for meaningful arithmetic and interpolation.

• Semantic Mutagenesis: Moving a sequence embedding in a specific direction can increase predicted thermostability while preserving fold.
• Interpolation: Linear interpolation between two functional enzymes can generate intermediate sequences with blended properties.
• This property is foundational for generative protein design using models like ProtGPT2 and ProGen2.

A single pre-trained embedding model can be applied to diverse prediction tasks without fine-tuning. The quality of the representation determines downstream performance.

• Fitness landscape reconstruction: Embedding distances correlate with functional similarity.
• Variant effect scoring: The log-likelihood ratio between wild-type and mutant embeddings (perplexity) predicts deleteriousness.
• This contrasts with task-specific models that require labeled training data for every new assay.

A self-supervised pre-training objective where random amino acids in a sequence are masked and the model learns to predict them from the surrounding context. This forces the model to learn rich, contextual embeddings.

• Process: 15% of residues are randomly masked; model predicts original amino acid
• Result: Embeddings capture local and global sequence dependencies
• Analogy: Identical to BERT's training objective, adapted for amino acid alphabets
• Benefit: No labeled data required—learns from raw sequences alone

A metric derived from language models that quantifies how surprising or unlikely a given amino acid sequence is under the model's learned distribution. Lower perplexity indicates sequences that are more 'natural' to the model.

• Formula: Exponential of the average negative log-likelihood per token
• Use case: Assessing sequence quality, detecting misfolded designs, variant effect ranking
• Interpretation: High perplexity at a position signals a potentially destabilizing or non-functional mutation

The process of navigating a protein language model's latent embedding space to generate novel sequences with altered properties by interpolating or perturbing learned representations.

• Techniques: Latent space interpolation, gradient-based optimization, directed evolution in silico
• Goal: Generate sequences with enhanced stability, altered binding, or novel functions
• Foundation: Relies on the smooth, continuous nature of learned protein embeddings where similar sequences cluster together