Graph Neural Network Embedding

Unlike matrix factorization which only captures direct interactions, GNN embeddings propagate signals across multiple hops in the interaction graph. A user's representation is influenced not just by items they clicked, but by items clicked by other users with similar behavior. This is achieved through graph convolutional layers that iteratively aggregate neighbor information, allowing the model to capture collaborative signals beyond the 1-hop neighborhood. The depth of propagation (number of layers) determines the receptive field.

The core operation is neighborhood aggregation. In each layer, a node (user or item) receives 'messages' from its connected neighbors. These messages are transformed via a learnable function and combined using an aggregation operator:

• Aggregators: Mean, Sum, Max, or attention-based (GAT)
• Combination: The aggregated neighbor vector is merged with the node's own previous representation
• Non-linearity: A transformation (e.g., linear layer + ReLU) is applied to produce the new embedding This process is repeated for L layers, synthesizing information from the L-hop neighborhood.

The definition of the graph structure critically impacts embedding quality. Common constructions include:

• Bipartite User-Item Graph: The standard setup with edges representing interactions (clicks, purchases). Edges can be weighted by interaction strength or recency.
• Item-Item Co-occurrence Graph: Items are connected if they appear in the same session or user history, enabling item-to-item recommendations without user nodes.
• Knowledge Graph Integration: External knowledge graph triplets (item-attribute-entity) are fused into the graph, providing semantic side information that mitigates cold-start issues.
• Temporal Graphs: Edges include timestamps, and message passing is constrained to respect chronological order, preventing information leakage from future interactions.

A critical advantage over transductive methods (like standard matrix factorization) is the ability to generate embeddings for unseen nodes at inference time. Because GNNs learn an aggregation function parameterized by the node's features, not a direct embedding lookup, they can compute a vector for a new user or item as soon as its initial features and connections are established. This directly addresses the cold-start problem by leveraging content features (e.g., item description, user demographics) through the graph convolution layers.

GNN architectures are broadly categorized by their mathematical foundation:

• Spectral Methods (e.g., Spectral CNN, ChebNet): Define convolution via the graph Laplacian's eigendecomposition in the Fourier domain. They are theoretically grounded but computationally expensive and inherently transductive, as filters are tied to the specific graph structure.
• Spatial Methods (e.g., GraphSAGE, GAT, GIN): Define convolution directly on the graph topology by aggregating features from spatial neighbors. These are more flexible, scalable, and support inductive learning, making them the dominant choice for modern recommender systems like PinSage at Pinterest.

As the number of graph convolutional layers increases, node representations tend to converge to indistinguishable vectors—a phenomenon known as over-smoothing. This occurs because repeated averaging of neighbor features washes out local information. Mitigation strategies include:

• Skip Connections: Adding the input of a layer to its output (residual connections) preserves original feature signals.
• Layer Normalization: Stabilizes training and prevents feature collapse.
• DeeperGCN Architectures: Employ specialized residual gating mechanisms.
• DropEdge: Randomly removing edges during training acts as a data augmentation that slows the convergence of node representations.

The core computational mechanism of GNNs where nodes iteratively aggregate feature information from their local neighborhood. In a user-item graph, a user node receives transformed feature vectors from connected item nodes, and vice versa. Each layer typically involves three steps: message computation (transforming neighbor features), aggregation (using permutation-invariant functions like mean, sum, or max pooling), and update (combining aggregated messages with the node's own previous state via a neural network layer). This process captures high-order connectivity—a user can receive information from an item they never directly interacted with, propagated through intermediate nodes.

A simplified graph convolution architecture that removes feature transformation and non-linear activation layers from standard GCNs, retaining only the neighborhood aggregation component. Key characteristics:

• Eliminates weight matrices and ReLU activations between propagation layers
• Uses only the normalized sum of neighbor embeddings at each layer
• Final embedding is the weighted sum of layer-wise embeddings
• Achieves superior performance on collaborative filtering benchmarks while being computationally lighter
• Demonstrates that non-linearity is unnecessary for collaborative filtering when the only features are learnable IDs

The ability of multi-layer GNNs to model relationships beyond direct interactions. A 1-hop connection links a user to items they've engaged with. A 2-hop connection links a user to other users who interacted with the same items (collaborative signal). A 3-hop connection links a user to items those similar users engaged with. This implicit collaborative filtering signal is automatically encoded into the embedding through iterative propagation. The number of GNN layers determines the maximum hop distance captured, though over-smoothing can occur with too many layers, causing all node embeddings to converge to similar values.

The process of building the bipartite or heterogeneous graph that serves as input to the GNN. For user embedding generation, the graph is typically:

• Bipartite: Two disjoint node sets (users and items) with edges representing interactions (clicks, purchases, ratings)
• Heterogeneous: May include additional node types (brands, categories, sessions) and edge types (viewed, added-to-cart, purchased)
• Edge-weighted: Interaction strength can be encoded as edge weights (purchase > click)
• Temporal: Edges may include timestamps for time-aware propagation
• Dynamic: The graph structure evolves as new interactions stream in, requiring incremental update strategies

An industrial-scale GNN framework developed by Pinterest that introduced key innovations for production deployment:

• Importance-based sampling: During training, not all neighbors are aggregated; a random walk-based importance score selects the most influential neighbors
• MapReduce-style minibatching: Constructs localized computation graphs centered on target nodes for efficient GPU training
• Curriculum training: Gradually increases the neighborhood size during training for stability
• Hard negative mining: Selects challenging negative samples that are close to the query in embedding space
• Deployed on a graph with 3 billion nodes and 18 billion edges for related pin recommendations

A degradation phenomenon in deep GNNs where node embeddings become indistinguishable as the number of propagation layers increases. As each layer aggregates information from a wider neighborhood, the receptive field of every node expands to cover most of the graph, causing all embeddings to converge to a similar value. Mitigation strategies include:

• Residual connections that preserve the node's initial embedding through skip connections
• Layer-wise weighted summation (as in LightGCN) rather than using only the final layer
• DropEdge regularization that randomly removes edges during training
• Limiting depth to 2-4 layers for most collaborative filtering tasks