Invariant Point Attention (IPA)

IPA is designed to be SE(3) equivariant. This means if the input 3D coordinates are rotated or translated, the attention logits remain invariant (unchanged), while the output coordinates transform consistently. This is achieved by operating on local coordinate frames attached to each residue, rather than raw global coordinates. The algorithm uses Euclidean distance and relative spatial transformations (rotations and translations) between these frames, which are inherently invariant to the global frame of reference. This eliminates the need for data augmentation by random rotations during training.

The IPA module computes attention weights using a combination of three terms:

• Scalar product attention: A standard query-key dot product on the abstract sequence and pair representations.
• Distance-based attention: A learned function of the Euclidean distance between residue centers, enforcing that spatially proximate residues attend to each other.
• Orientation-based attention: A learned function of the relative rotation matrix between residue frames, capturing the geometric compatibility of local coordinate systems. These terms are summed to produce the final attention logits, which are then used to weight the update of the residue representations.

IPA is not applied once, but iteratively within the Structure Module. The module takes the abstract representations from the Evoformer and initializes a set of residue frames as a black hole initial guess (all frames at the origin). Over 8 recycling iterations, IPA updates both the abstract representations and the 3D coordinates of each residue frame. Each iteration produces a more refined structure, with the final output being the predicted all-atom coordinates. This iterative process allows the model to resolve complex topological arrangements step-by-step.

Unlike standard multi-head attention in transformers, which operates on a fully connected graph of sequence positions, IPA operates on a sparse, geometry-aware graph. The attention is computed on a set of 3D points (the residue centers) with associated frames (rotation matrices). This allows the model to reason about spatial relationships like bond angles, steric clashes, and hydrogen bonding patterns directly in 3D space. The 'Invariant Point' refers to the fact that the attention mechanism uses a set of learned 3D query, key, and value points projected from each residue's local frame, which are invariant to global transformations.

A naive implementation of 3D point attention would be computationally prohibitive for proteins with thousands of residues. IPA achieves efficiency through several design choices:

• Linear complexity in the number of points: The attention is computed over a fixed number of invariant points per residue, not all atoms.
• Shared geometry: The relative spatial transformations are computed once and shared across all attention heads.
• Recycling: The iterative refinement allows the model to use a relatively small number of IPA blocks (8) to achieve high accuracy, rather than a single, massive forward pass. This design makes the Structure Module tractable for typical protein lengths.

IPA predicts the protein backbone as a series of rigid-body frames, each defined by a 3D translation vector and a 3x3 rotation matrix. The backbone is parameterized by three dihedral angles (φ, ψ, ω) and the Cα position. After the backbone is predicted, the side-chain torsion angles (χ angles) are predicted by a separate network that takes the IPA-refined representations as input. This two-stage process—backbone via IPA, side-chains via a subsequent network—mirrors the hierarchical nature of protein structure and ensures the backbone geometry is physically valid before side-chain packing.