Graph Attention Network (GAT)

Unlike Graph Convolutional Networks (GCNs) which use fixed, static weights derived from the graph's Laplacian, GATs dynamically compute attention coefficients for every edge during each forward pass. A shared, learnable linear transformation is applied to node features, followed by a self-attention mechanism that outputs a scalar e_ij representing the raw importance of node j's features to node i. This allows the model to implicitly assign different weights to different neighbors without requiring prior knowledge of the graph structure or expensive matrix inversions.

To stabilize the learning process of self-attention, GATs employ a multi-head attention mechanism popularized by the Transformer architecture. The paper's authors found that the attention scores could exhibit high variance.

• K independent attention heads execute the attention mechanism in parallel.
• Their outputs are concatenated in hidden layers to produce enriched feature representations.
• At the final prediction layer, outputs are averaged instead of concatenated.

This ensemble-like approach ensures the model captures diverse relational semantics and reduces reliance on any single noisy attention head.

GATs perform masked attention, meaning the attention mechanism is computed only for a node's immediate first-order neighbors (including a self-loop). This injects the graph's structural information directly into the computation. The softmax normalization is applied locally to the coefficients of a node's neighborhood, ensuring that attention weights sum to 1 across all neighbors. This structural masking makes the architecture inductive, capable of generalizing to entirely unseen nodes and graphs, a critical limitation of transductive methods like the original GCN.

The self-attention layer operates on node-feature pairs independently, with no expensive matrix eigendecomposition or operations on the full adjacency matrix. This makes GATs highly efficient compared to spectral methods.

• Parallelizable: Attention coefficients for all edges can be computed simultaneously.
• Scalable: The operation is a local neighborhood aggregation, making it suitable for large, dense graphs.
• No global context needed: The model does not require the entire graph to be loaded into memory for a forward pass, enabling mini-batch training on massive graph datasets.

While the original GAT formulation computes attention based solely on node features, the mechanism naturally extends to incorporating edge features. The attention coefficient e_ij can be modified to accept a concatenation of the source node, destination node, and their shared edge attributes. This allows the model to learn that certain relationship types (e.g., 'contraindicates' vs. 'treats' in a medical knowledge graph) are inherently more critical for a specific predictive task, providing a richer, relation-aware message-passing scheme.

A significant advantage over black-box graph models is the inherent interpretability of GATs. After training, the calculated attention coefficients α_ij can be extracted and analyzed. A high attention weight from a drug node to a disease node provides an explicit, quantitative explanation for the model's prediction. This allows clinical informaticists and data scientists to audit the model's reasoning, verifying that it focuses on medically relevant relationships rather than spurious correlations, which is critical for clinical decision support systems.

The parent class of deep learning architectures that operate on graph-structured data. GNNs learn node representations through a recursive message-passing scheme: each node aggregates feature information from its neighbors, updates its own state, and passes messages outward. This iterative process captures both local topology and node attributes. GATs extend this paradigm by introducing learnable attention coefficients to weight neighbor importance during aggregation, rather than using fixed weights based on graph structure alone.

A foundational GNN variant that generalizes the convolution operation to irregular graph domains. GCNs aggregate neighbor features using normalized adjacency-based weights derived from the graph Laplacian, giving equal or degree-scaled importance to all neighbors. The key distinction from GATs: GCN weights are fixed by graph structure, while GATs compute dynamic, content-aware attention scores. This makes GATs more flexible for heterogeneous graphs where not all neighbors are equally relevant.

The core computational primitive that GATs borrow from the Transformer architecture. Self-attention computes a weighted representation of an input sequence by allowing each element to attend to all others, with weights determined by pairwise compatibility scores. In GATs, this mechanism operates over a node's neighborhood set: for node i and neighbor j, an attention coefficient α_ij is computed via a shared linear transformation followed by a LeakyReLU-activated dot product with a learnable attention vector. Multi-head attention stabilizes learning.

A unifying framework that formalizes the common structure across GNN variants. MPNNs define two differentiable functions:

• Message function: Computes messages from neighbor nodes' features and edge attributes
• Update function: Aggregates incoming messages and updates the node's hidden state GATs fit this framework by using attention-weighted message aggregation, where the message is the neighbor's transformed feature vector and the aggregation is a weighted sum with learned attention coefficients. This allows the model to implicitly learn edge weights.

The low-dimensional, dense vector representation of a graph node that encodes its structural position, local neighborhood properties, and feature information. GATs produce contextualized node embeddings where the representation of a node is influenced more heavily by important neighbors as determined by attention scores. These embeddings serve as input to downstream tasks:

• Node classification: Predict labels for individual nodes
• Link prediction: Estimate probability of edges between node pairs
• Graph classification: Aggregate node embeddings for whole-graph prediction

A technique employed by GATs to stabilize the learning process and capture diverse relational patterns. Instead of computing a single attention distribution, K independent attention mechanisms operate in parallel, each with its own parameters. Their outputs are either:

• Concatenated (intermediate layers): Produces K×d' dimensional features
• Averaged (final prediction layer): Maintains consistent output dimensionality This ensemble approach reduces variance from the random initialization of attention parameters and allows different heads to specialize in different types of neighbor relationships.