Graphormer

A learnable embedding vector assigned to each node based on its degree centrality (number of neighbors). This encoding captures the node's importance in the molecular graph.

• In-degree and out-degree are encoded separately for directed graphs
• Allows the model to distinguish between terminal atoms (e.g., hydrogen) and highly connected central atoms (e.g., carbon in a ring system)
• The encoding is added to the input token embedding before the first attention layer
• Critical for tasks where atom connectivity correlates with reactivity, such as predicting sites of metabolic oxidation

A scalar bias term added to the attention score between node pairs, derived from the shortest path distance (SPD) between them in the molecular graph. This is not a simple lookup; it is a learnable scalar indexed by distance.

• If two atoms are not connected, a special distance token is used
• Encoded as a bias function b(SPD(i,j)) that modulates the pre-softmax attention logits
• Allows the attention mechanism to weigh information based on topological proximity
• Replaces the need for positional encodings used in sequence Transformers, as molecular geometry is graph-defined

Encodes multi-dimensional edge features along the shortest path between two nodes into a single bias vector added to their attention calculation.

• For each node pair (i, j), the shortest path SP_ij = (e_1, e_2, ..., e_N) is identified
• The edge features (e.g., bond type, stereochemistry) along this path are averaged after a learnable dot-product with a weight vector
• The resulting scalar is added to the attention score
• This allows the model to understand that the relationship between two atoms depends on the types of bonds connecting them, not just their distance

The standard Transformer self-attention is augmented by summing the spatial encoding and edge encoding biases into the Query-Key dot product before the softmax normalization.

• The attention formula becomes: A = softmax( (QK^T)/√d + b_spatial + b_edge )
• This is a plug-and-play modification to any standard Transformer block
• Each attention head can learn to attend to different structural patterns independently
• Enables the model to capture long-range dependencies in large molecules, such as protein-ligand interactions, without the depth limitations of message-passing networks

A virtual super-node connected to all atoms in the graph, serving as a global readout mechanism. The [VNode] embedding is updated at each layer by aggregating information from all real nodes.

• Connected to every atom with a learnable spatial bias, typically a distinct b(SPD_VNode) value
• Aggregates a graph-level representation without a separate pooling function
• Its final embedding is used for graph-level prediction tasks, such as predicting molecular energy or solubility
• Eliminates the information bottleneck common in standard graph pooling operations

An extension that incorporates 3D atomic coordinates into the spatial encoding by using the Euclidean distance between atoms, rather than just the graph-theoretic shortest path.

• The spatial bias becomes a function of the continuous Euclidean distance, often binned or processed via a Gaussian kernel
• Allows the model to reason about conformer-specific properties like steric hindrance and non-bonded interactions
• Bridges the gap between 2D topological graphs and 3D geometric deep learning models
• Achieves competitive performance on the PCQM4Mv2 benchmark without explicit 3D message passing

A core architectural innovation that injects graph structural priors directly into the attention mechanism. Graphormer assigns each node a learnable embedding vector based on its degree centrality (in-degree and out-degree), which is added to the input features. This allows the model to distinguish between a carbon atom with two neighbors and one with four, encoding the local topology without requiring a separate graph neural network. This simple yet powerful technique is key to surpassing standard GNNs on molecular benchmarks.

A mechanism that biases the self-attention computation using the shortest path distance (SPD) between node pairs. Unlike positional encodings in NLP, Graphormer uses a learnable scalar bias term for each discrete distance. If two atoms are connected by a path of length 3, a specific bias is added to their attention score. This captures the global molecular geometry and long-range dependencies, enabling the model to understand that functional groups on opposite ends of a molecule can still interact electronically.

A method for incorporating multi-dimensional edge features into the attention layer. For each node pair, Graphormer computes the dot product of edge features along the shortest path connecting them, averaged and transformed by a learnable function. This allows the model to process rich bond information—such as bond type (single, double, aromatic), stereochemistry, and conjugation—directly within the transformer's core operation, rather than treating edges as secondary data.

An extension of the architecture that integrates three-dimensional atomic coordinates for tasks requiring spatial awareness. The spatial encoding is modified to use Euclidean distance instead of graph distance, and the model incorporates pairwise 3D vectors to capture angular relationships. This variant achieves state-of-the-art results on quantum mechanical property prediction benchmarks like QM9 and MD17, accurately predicting energies and forces by directly modeling the continuous 3D conformation of molecules.

A specialized application where Graphormer serves as the molecular encoder in a template-free retrosynthesis pipeline. The model encodes the target molecule into a rich latent representation, which is then decoded by a transformer to generate reactant SMILES strings. By leveraging centrality and spatial encodings, the model excels at identifying reaction centers—the specific bonds that should be disconnected—and generating valid synthons, outperforming sequence-only models that lack explicit structural awareness.

A resilient variant designed to handle graph noise and distribution shift, common in real-world molecular data. Graphormer-GD introduces a graph denoising objective during pre-training, where the model learns to reconstruct corrupted structural information. This self-supervised task forces the model to develop robust internal representations of molecular topology, making it significantly more accurate when applied to downstream tasks with noisy or incomplete experimental data, such as bioactivity prediction from high-throughput screening.