Self-Attention

Unlike recurrent neural networks (RNNs) that process tokens sequentially, self-attention computes interactions between all positions in a sequence simultaneously. This parallelization dramatically reduces training time and mitigates the vanishing gradient problem for long-range dependencies. Each input token attends to every other token in a single matrix multiplication, making the mechanism highly efficient on modern GPU hardware.

The mechanism operates by projecting each input vector into three distinct representations:

• Query (Q): The token seeking information, asking 'What is relevant to me?'
• Key (K): The token acting as an index, advertising 'This is what I represent.'
• Value (V): The actual content to be aggregated if the key matches the query. The dot product between a Query and all Keys produces an attention score matrix, which weights the aggregation of Values.

The raw attention scores are computed as the dot product of the Query and Key matrices, then scaled down by the square root of the key dimension (√dₖ). This scaling factor prevents the softmax function from entering regions with extremely small gradients when the dot products grow large in high-dimensional space. Without scaling, the gradients would vanish, halting learning. The formula is: Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V.

Rather than performing a single attention function, the model projects Q, K, and V into multiple lower-dimensional subspaces (heads) in parallel. Each head learns to attend to different representational relationships:

• One head may track syntactic dependencies.
• Another may resolve coreference between pronouns and entities.
• A third may capture long-range semantic associations. The outputs of all heads are concatenated and linearly projected, allowing the model to jointly attend to information from different representation subspaces.

Self-attention is inherently permutation-invariant—it has no built-in notion of token order. To inject sequence information, sinusoidal positional encodings or learned position embeddings are added to the input embeddings before the first attention layer. These encodings allow the model to distinguish between 'The dog bit the man' and 'The man bit the dog' by providing a unique signal for each position, enabling the attention mechanism to factor in relative distances.

A single self-attention layer provides every token with a direct connection to every other token in the sequence, regardless of distance. In a 500-token sequence, the first and last tokens interact in a single step. This contrasts sharply with RNNs, which require O(n) sequential steps, and CNNs, which require stacking many layers to expand the receptive field. This property makes self-attention exceptionally effective for capturing long-range dependencies in user behavior sequences and session data.

A technique that injects information about token position into the self-attention mechanism, which is inherently permutation-invariant. Without it, the model cannot distinguish "cat ate mouse" from "mouse ate cat."

• Sinusoidal encodings: Fixed functions of position and dimension
• Learned embeddings: Trainable position vectors
• Relative positional encoding: Captures pairwise distances between tokens

An extension of self-attention that runs multiple attention operations in parallel, allowing the model to attend to information from different representation subspaces simultaneously.

• Each head learns distinct relational patterns
• Head 1 might capture syntactic dependencies
• Head 2 might focus on semantic similarity
• Outputs are concatenated and linearly projected

The core mathematical operation: Attention(Q,K,V) = softmax(QK^T / √d_k)V. The scaling factor 1/√d_k prevents the dot products from growing too large, which would push the softmax into regions with extremely small gradients.

• Q (Query): What the token is looking for
• K (Key): What the token can match against
• V (Value): The information to aggregate

An architecture that uses attention mechanisms to adaptively compute user interest representations from historical behaviors. Unlike self-attention over the full sequence, DIN activates only relevant historical interactions given a specific candidate item.

• Computes candidate-aware attention weights
• Suppresses irrelevant past behaviors
• Enables adaptive interest modeling at inference time