Graph Attention Network (GAT)

The core innovation of GATs is the computation of attention coefficients (α_ij) that implicitly assign different importance to every edge in the neighborhood. Unlike Graph Convolutional Networks (GCNs) which use fixed, structure-based weights, GATs compute these coefficients via a shared self-attention mechanism. A learnable weight matrix W is applied to every node, followed by a shared attentional mechanism a (a single-layer feedforward network) that computes raw attention scores e_ij = LeakyReLU(a^T [Wh_i || Wh_j]). These scores are then normalized across all neighbors using a softmax function, making the coefficients comparable across different neighborhoods and invariant to neighborhood size.

GATs stabilize the learning process by employing multi-head attention, inspired by the Transformer architecture. Instead of computing a single attention function, the model runs K independent attention mechanisms in parallel. The outputs are then combined:

• Concatenation: In hidden layers, the K feature vectors are concatenated, producing a richer, higher-dimensional representation.
• Averaging: In the final prediction layer, the outputs are averaged, which is more appropriate for classification tasks. This parallelism reduces the variance caused by the non-deterministic nature of attention weight initialization and allows the model to jointly attend to different representation subspaces.

The self-attention layer in GATs operates on node-feature pairs, making the computation highly parallelizable across all edges of the graph simultaneously. Unlike spectral methods, GAT does not require an eigen-decomposition of the Laplacian matrix. The operation has a time complexity of O(|V|FF' + |E|F'), where V is nodes, E is edges, and F is features. Crucially, GAT is an inductive model: because the attention weights are a function of node features rather than the global graph structure, a trained model can be directly applied to entirely unseen graphs without retraining, a critical requirement for dynamic molecular libraries.

GATs replace the static, non-parametric aggregation of GCNs with a dynamic, feature-dependent weighting function. The final representation for a node i is a weighted sum of transformed neighbor features: h'_i = σ( Σ α_ij W h_j ). This allows the model to automatically learn to:

• Ignore noisy edges: Assign near-zero weight to irrelevant or spurious connections.
• Focus on functional groups: In molecular graphs, a node can learn to attend strongly to specific chemical moieties (e.g., an electrophilic carbon attending to a nucleophilic group) regardless of their topological distance.
• Handle heterophily: Unlike GCNs which implicitly assume homophily, GATs can learn to weigh dissimilar neighbors appropriately.

To preserve the inductive bias of the input graph topology, GATs inject the structural information via masked attention. Unlike the Transformer where every token attends to every other token, GAT computes e_ij only for nodes j in the closed neighborhood N_i of node i (including a self-loop). This masking ensures the model respects the molecular connectivity while still learning which specific bonds are most critical for the target property. The self-loop α_ii is a learned parameter that determines how much of a node's own features to retain, preventing the model from being overly dominated by neighbors.

A key theoretical limitation of the original GAT is that it computes static attention: the ranking of attention scores is global and independent of the query node. For any set of keys, the attention function a is monotonic, meaning the highest-scoring neighbor will always be the same regardless of the query node. This limits the model's ability to capture complex, context-dependent interactions. The Graph Isomorphism Network (GIN) has been shown to be strictly more powerful than GAT in distinguishing non-isomorphic graphs. Subsequent architectures like GATv2 fix this by applying the weight matrix W after the concatenation, enabling dynamic, query-dependent attention.