Evoformer

Processes the MSA representation by applying self-attention along the sequence dimension for each aligned sequence independently. This operation allows the model to identify coevolving residue pairs within individual sequences. A gating mechanism controls the flow of information, allowing the network to ignore uninformative or gapped alignment columns, which is critical for handling the variable quality of MSAs.

Operates on the MSA representation by applying attention along the depth of the alignment, across all sequences for a single residue position. This allows the model to assess the conservation patterns of specific amino acids across the protein family. By analyzing which residues are conserved or variable at a given position, the network infers evolutionary constraints and functional importance.

The core information bottleneck that connects the MSA and pair representations. It computes the outer product of the MSA representation vectors for every residue pair, averages this over all sequences, and projects the result into the pair representation. This operation transforms evolutionary correlation signals from the MSA into pairwise structural constraints, effectively encoding residue coevolution into a geometric prior.

Enforces the triangle inequality on the pair representation, a fundamental geometric constraint of 3D space. For any three residues i, j, and k, the distance between i and j cannot exceed the sum of distances between i-k and k-j. This operation uses multiplicative gating to iteratively refine pairwise edge features, ensuring the learned representations are consistent with a valid 3D structure before any coordinates are generated.

A complementary mechanism to the multiplicative update that also enforces geometric consistency. It computes attention over triplets of residues (i, j, k) to update the edge between i and j. By starting from one node and attending to all possible third nodes, the model can reason about the global structural context of each residue pair, ensuring that local pairwise predictions are coherent with the overall protein topology.

Standard point-wise feed-forward layers applied independently to each element of the MSA and pair representations after each attention or communication block. These use a SwiGLU activation and expand the channel dimension by a factor of 4 before projecting back. They provide the non-linear processing capacity needed to refine the features extracted by the attention mechanisms, analogous to the feed-forward networks in standard Transformer architectures.

An N x N matrix representing the relationship between every pair of residues in the target protein. Initialized with features like relative distance and orientation from templates, this representation is updated by the Evoformer's triangular multiplicative updates. It encodes the predicted inter-residue distances and orientations, effectively acting as a learned, implicit energy potential that guides the final structure generation.

The defining mechanism of the Evoformer is the continuous, bidirectional flow of information between the MSA representation and the pair representation. The model uses a series of operations to project MSA sequence correlations into the pair space, and conversely, uses the pair representation to gate and refine the MSA attention. This ensures that raw evolutionary statistics are constrained by geometric plausibility.

A key operation applied to the pair representation that enforces the triangle inequality on predicted distances. For any three residues (i, j, k), the distance between i and k cannot exceed the sum of distances between i and j and j and k. This geometric constraint is baked into the architecture using outgoing and incoming triangular gates, ensuring the predicted pair representation corresponds to a valid 3D structure.

While not part of the Evoformer itself, the Structure Module consumes the Evoformer's output. The IPA is the Structure Module's core algorithm, acting as a form of SE(3)-equivariant attention. It uses the refined pair representation to attend over 3D point clouds of residue frames, iteratively refining the protein backbone geometry while remaining invariant to global rotation and translation.

The biological principle that makes the Evoformer effective. Residue pairs that are in close contact in the 3D structure tend to mutate in a correlated fashion over evolutionary time to maintain structural integrity. The Evoformer's attention mechanisms are explicitly designed to detect these correlated mutation patterns in the MSA and translate them into spatial proximity constraints in the pair representation.