Equiformer

Replaces standard scalar dot-product attention with an SE(3)/E(3)-equivariant attention operation. Instead of computing attention weights solely from scalar queries and keys, Equiformer incorporates higher-order tensor products of directional information (e.g., relative position vectors). This allows the attention mechanism to be modulated by the 3D geometry of the molecule, ensuring that the attention weights themselves transform correctly under rotation and translation.

• Uses Clebsch-Gordan tensor products to combine geometric features
• Attention weights are invariant, but value transformations are equivariant
• Preserves directional information across layers without information loss

Adapts the standard Transformer architecture by replacing linear layers with equivariant tensor product operations. Instead of operating on scalar vectors alone, Equiformer processes features as irreducible representations (irreps) of the SO(3) group. Each layer updates both scalar (type-0) and higher-order tensor features (type-1, type-2, etc.) using learned weights, ensuring all operations are strictly equivariant.

• Multi-head attention operates on multi-channel irreps
• Layer normalization is adapted for equivariant features
• Depth-wise tensor products reduce computational overhead

Equiformer achieves leading performance on the QM9 quantum chemistry benchmark and the Open Catalyst Project (OC20) dataset. On QM9, it sets new records for predicting properties like HOMO-LUMO gap, dipole moment, and internal energy. On OC20, it demonstrates superior ability to predict the adsorption energy of molecules on catalyst surfaces, a critical task for renewable energy research.

• QM9: Mean absolute error (MAE) for HOMO-LUMO gap reduced to ~20 meV
• OC20 S2EF: Improved force prediction accuracy by ~15% over prior equivariant models
• Data-efficient: achieves high accuracy with fewer training examples than non-equivariant baselines

Unlike message-passing GNNs that suffer from over-smoothing with increased depth, Equiformer's Transformer architecture allows for deeper networks (8-12 layers) without degradation. The self-attention mechanism provides global receptive fields, enabling each atom to attend to all others directly. Combined with higher-order tensor features (up to l=3 or l=4), this yields exceptional expressivity for modeling complex quantum interactions.

• Global attention avoids the locality bottleneck of message passing
• Higher-order irreps capture angular dependencies precisely
• Depth scaling improves accuracy without diminishing returns

Equiformer provides a mathematical guarantee that predictions are invariant to global rotation and translation (and optionally reflection). This is achieved by constructing all operations—attention, feed-forward networks, and normalization—from equivariant building blocks. The output is a set of invariant scalar features suitable for property prediction, while intermediate layers preserve full geometric tensor information.

• SE(3): Rotation and translation equivariance (no reflection)
• E(3): Includes reflection equivariance for parity-sensitive properties
• Guarantees physical consistency without data augmentation

Equiformer bridges the gap between two families of equivariant models:

• vs. EGNN: EGNN uses only vector features (type-1) for efficiency, while Equiformer employs higher-order tensor products (type-2, type-3) for greater expressivity, achieving higher accuracy on complex tasks.
• vs. NequIP: NequIP is a message-passing network with tensor products; Equiformer replaces message passing with self-attention, enabling global context and better scaling to large molecular systems.
• Trade-off: Equiformer is more computationally intensive than EGNN but more expressive; it is more scalable than NequIP for large graphs due to attention's parallelizability.

The core mathematical principle behind Equiformer. A function is SE(3) equivariant if rotating and translating the input 3D structure produces an identically transformed output. This ensures the model's predictions are independent of a molecule's orientation in space, a critical inductive bias for learning physical properties. Equiformer achieves this through tensor products of irreducible representations, unlike scalar-only models that lose directional information.

A foundational architecture that introduced learned, locally equivariant filters operating on point clouds. It maps geometric features to higher-order tensor fields using filters conditioned on relative positions. Equiformer extends this concept by integrating tensor product operations directly into the attention mechanism, replacing the standard scalar dot-product attention with an equivariant cross-product that preserves directional information between nodes.

A computationally efficient alternative to Equiformer that achieves E(n) equivariance without expensive higher-order tensor products. EGNN operates directly on scalar and vector features, updating coordinates as a weighted sum of relative distances. While faster, EGNN does not capture higher-order angular interactions between triplets of atoms, which Equiformer models through tensor product-based attention, giving it superior expressiveness for complex molecular geometries.

An E(3)-equivariant interatomic potential that, like Equiformer, uses tensor products of irreducible representations for message passing. NequIP demonstrated that equivariant models achieve data efficiency orders of magnitude better than invariant networks. Equiformer builds on this insight by replacing NequIP's message-passing framework with a transformer backbone, enabling global attention across the entire molecular graph rather than local neighborhood aggregation.

A standard transformer adapted for graphs by encoding structural priors like node centrality and shortest-path distances into the attention bias. Graphormer achieved strong results on the Open Catalyst Project (OC20) benchmark but treats 3D coordinates as scalar distances, losing directional information. Equiformer surpasses this by making the attention mechanism itself SE(3)-equivariant, preserving full geometric tensor information throughout the network.

A generative framework that learns to reverse a noising process on 3D atomic coordinates while maintaining SE(3) equivariance. EDMs are used for tasks like conformer generation and structure-based drug design. The denoising backbone of an EDM can be instantiated with an Equiformer architecture, combining Equiformer's state-of-the-art property prediction accuracy with diffusion-based generative capabilities for molecular structure sampling.