Transformer Architecture

The core computational unit that allows each position in a sequence to attend to all other positions simultaneously. For each token, the model computes Query (Q), Key (K), and Value (V) vectors through learned linear projections. The attention weight between token i and token j is derived from the scaled dot-product of Q_i and K_j, determining how much of V_j contributes to the output representation of token i.

• Scaled Dot-Product Attention: Attention(Q,K,V) = softmax(QK^T / √d_k)V
• The scaling factor √d_k prevents the softmax from entering regions of extremely small gradients
• Enables direct modeling of dependencies regardless of their distance in the sequence
• Contrasts sharply with RNNs, where information must flow sequentially through hidden states

Rather than performing a single attention function, the Transformer projects Q, K, and V h times with different learned linear projections, performing attention in parallel across multiple representation subspaces. Each head can learn to attend to different types of relationships—syntactic structure, semantic similarity, or positional proximity—simultaneously.

• Typical configurations use 8 to 16 heads
• Outputs of all heads are concatenated and projected back to the model dimension
• Formula: MultiHead(Q,K,V) = Concat(head_1, ..., head_h)W^O
• Each head operates on a reduced dimension d_k = d_model / h to keep total computational cost constant

Since the Transformer contains no recurrence or convolution, it has no inherent notion of token order. Positional encodings inject sequence order information by adding a unique vector to each input embedding. The original paper used sinusoidal functions of varying frequencies:

• PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
• PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
• This allows the model to extrapolate to sequence lengths unseen during training
• Learned positional embeddings are a common alternative, treating position as a trainable lookup table
• Relative positional encodings (e.g., Rotary Position Embedding (RoPE)) encode the relative distance between tokens, improving length generalization

Each encoder and decoder layer contains a fully connected position-wise feed-forward network applied identically and independently to every token position. This component introduces non-linearity and increases the model's capacity to learn complex transformations.

• Consists of two linear transformations with a ReLU or GELU activation in between: FFN(x) = W_2 * activation(W_1 * x + b_1) + b_2
• The inner dimension is typically 4x the model dimension (e.g., 2048 for a 512-dim model)
• Can be viewed as performing a per-token key-value memory lookup
• SwiGLU and other gated variants have become standard in modern architectures like LLaMA and PaLM for improved training stability

Each sub-layer (attention and FFN) is wrapped with a residual connection followed by layer normalization. The residual connection adds the sub-layer's input directly to its output, creating a gradient highway that mitigates the vanishing gradient problem in deep networks.

• Post-LN: Normalization applied after the residual addition (original paper design)
• Pre-LN: Normalization applied to the input before the sub-layer, then added to the residual. Pre-LN has become dominant as it improves training stability and convergence speed
• The residual structure enables training Transformers with hundreds of layers
• Formula: Output = LayerNorm(x + Sublayer(x)) for Post-LN; Output = x + Sublayer(LayerNorm(x)) for Pre-LN

The original Transformer uses an encoder-decoder structure for sequence-to-sequence tasks like machine translation. The encoder stack processes the entire source sequence bidirectionally, while the decoder generates the target sequence autoregressively.

• Encoder: N identical layers, each with multi-head self-attention and FFN. All positions attend to all positions (bidirectional)
• Decoder: N identical layers with three sub-layers: masked self-attention (prevents attending to future positions), cross-attention (attends to encoder output), and FFN
• Cross-attention uses Q from the decoder and K,V from the encoder, allowing the decoder to focus on relevant source tokens at each generation step
• Modern decoder-only models (GPT, LLaMA) discard the encoder entirely, relying solely on causal self-attention for autoregressive generation

Self-attention computes a weighted representation of every item in a user's behavioral sequence by dynamically assessing the relevance of each past interaction to every other interaction. Unlike recurrent models that process actions step-by-step, self-attention allows the model to directly connect a purchase made 50 clicks ago to the current browsing context in a single operation.

• Query, Key, Value (QKV): Each item in the sequence is projected into three vectors. The attention weight between item i and item j is the scaled dot-product of the query for i and the key for j.
• Scaled Dot-Product Attention: The raw attention scores are divided by the square root of the dimension (d_k) to prevent vanishing gradients from large dot products in high-dimensional spaces.
• Parallelization: Because attention scores for all positions are computed simultaneously, Transformers train significantly faster on GPUs than LSTMs processing sequences token-by-token.

Since the Transformer has no inherent notion of sequence order, positional encodings are added to the input embeddings to inject information about the absolute or relative position of each user action. This is critical for retail sequences where the order of clicks—search, product view, add-to-cart, purchase—encodes the user's decision-making stage.

• Sinusoidal Encoding: The original Transformer uses fixed sine and cosine functions of varying frequencies, allowing the model to extrapolate to sequence lengths unseen during training.
• Learned Positional Embeddings: Many retail implementations, such as the Behavior Sequence Transformer (BST), learn a dedicated embedding vector for each position, which is trained jointly with the model.
• Relative Positional Encoding: Variants like Transformer-XL encode the distance between pairs of items, which is more natural for modeling the relative recency of a click versus an absolute timestamp.

Multi-head attention runs multiple self-attention operations in parallel, each with its own learned linear projections. This allows the model to simultaneously attend to different aspects of a user's behavioral sequence.

• Head 1 might learn to focus on short-term session intent, attending heavily to the last 3-5 clicks.
• Head 2 might capture long-term brand affinity, linking the current session to a purchase made weeks ago.
• Head 3 might identify price sensitivity patterns, correlating views of discounted items with cart abandonment.

The outputs of all heads are concatenated and projected, giving the subsequent layer a rich, multi-faceted representation of the user's sequential behavior.

After the multi-head attention sub-layer, each position's representation is passed through an identical position-wise feed-forward network (FFN). This consists of two linear transformations with a ReLU activation in between: FFN(x) = max(0, xW₁ + b₁)W₂ + b₂.

• Role: While attention mixes information across positions, the FFN applies a non-linear transformation independently to each position, increasing the model's capacity to learn complex feature interactions.
• Dimensionality: The inner hidden layer is typically 4x larger than the model dimension (d_ff = 2048 for d_model = 512), creating a bottleneck that forces the model to learn compressed, generalizable representations.
• Retail Relevance: For a sequence of product views, the FFN can learn non-linear interactions between a product's category embedding and its price embedding at each time step.

Each sub-layer (multi-head attention and feed-forward) is wrapped with a residual connection followed by layer normalization. This architectural choice is critical for training deep Transformers on long user sequences without degradation.

• Residual Connection: The input to a sub-layer is added to its output: LayerNorm(x + Sublayer(x)). This provides a direct gradient path backward, mitigating the vanishing gradient problem in deep stacks.
• Layer Normalization: Normalizes the activations across the feature dimension for each individual sequence position, stabilizing training and accelerating convergence.
• Pre-LN vs. Post-LN: Modern implementations often apply layer normalization before the sub-layer (Pre-LN), which has been shown to improve training stability for very deep models used in large-scale retail catalogs.