Behavioral Sequence Transformer

The core mechanism computes pairwise attention weights between every position in a user's action sequence. Unlike RNNs, which process steps sequentially, self-attention allows the model to directly connect a purchase made weeks ago with a current browsing session. Each action embedding is transformed into a Query, Key, and Value vector. The dot-product similarity between a Query and all Keys determines how much focus each past action receives when encoding the current context.

• Long-Range Dependencies: Captures signals like 'user bought a camera last month, now viewing lenses'.
• Parallel Computation: All positions are processed simultaneously during training, drastically reducing wall-clock time.

Since self-attention is permutation-invariant, the model must be explicitly told the order of actions. BSTs inject sinusoidal positional encodings or learned position embeddings into the input sequence. This allows the architecture to distinguish between 'viewed item A then B' versus 'viewed B then A'.

• Temporal Gap Awareness: Advanced implementations encode the time delta between actions, teaching the model that a click 5 seconds ago is contextually different from a click 5 days ago.
• Sequence Length Handling: Enables the model to generalize to longer user histories than seen during training.

A single attention function can only focus on one type of relationship. Multi-head attention projects the input into multiple subspaces, allowing the model to simultaneously attend to different aspects of a user's behavior.

• Short-Term vs. Long-Term: One head might focus on the immediate last-click session, while another tracks a user's long-term genre preferences.
• Price Sensitivity & Brand Affinity: Different heads can specialize in distinct behavioral dimensions without manual feature engineering.
• Ensemble Effect: The concatenated outputs from all heads provide a richer, more robust user representation.

Each attention layer is followed by a position-wise feed-forward network (two linear transformations with a non-linear activation like GELU). This component processes the attention-weighted context and introduces non-linearity. Residual connections wrap both the attention and feed-forward sub-layers, adding the input directly to the output before layer normalization.

• Stable Training: Residual paths allow gradients to flow unimpeded through deep stacks of transformer blocks, preventing vanishing gradients.
• Feature Refinement: The feed-forward network acts as a per-position feature refiner, transforming the aggregated context into a more expressive representation for the next layer.

The final layer of the BST extracts the output embedding corresponding to the last action in the sequence. This vector serves as a dynamic user embedding that encodes the user's current intent. It is fed into a scoring function—typically a dot product with a learned item embedding matrix—to produce a probability distribution over the entire item catalog.

• Self-Supervised Objective: The model is trained by masking the last item in a sequence and learning to predict it from the preceding actions.
• Efficient Retrieval: The output user embedding can be used directly in an Approximate Nearest Neighbor (ANN) index for sub-linear candidate generation.

To prevent information leakage from the future, a triangular attention mask is applied during training. This ensures that when encoding the action at position t, the model can only attend to positions 1 through t-1. This constraint makes the training objective a proper causal language modeling task, where the model learns to predict the next action based solely on the past.

• Realistic Inference: Mimics the production setting where only historical data is available.
• Bidirectional vs. Causal: Unlike BERT-style encoders, this masking strategy is essential for generative sequential recommendation.

Since self-attention is permutation-invariant, positional encoding injects information about the chronological order of actions. This allows the model to distinguish between a user who browsed shoes then socks versus socks then shoes.

• Sinusoidal Encoding: Adds fixed sine/cosine waves of different frequencies to the input embeddings, allowing the model to easily learn relative positions.
• Learned Positional Embeddings: Treats the absolute position index as a vocabulary item, learning a unique vector for each time step during training.
• Key Insight: Temporal dynamics are critical; a click on a winter coat in July has a vastly different intent signal than the same click in December.

The dynamic user representation generated by the transformer that encodes evolving intent and short-term session context. Unlike static user embeddings, this vector changes with every new action.

• Temporal Evolution: The embedding at time t captures the user's immediate goal, not just their long-term average preference.
• Contrastive Pre-training: Often trained using InfoNCE loss to pull the sequence embedding close to the embedding of the actual next-clicked item while pushing it away from random negatives.
• Integration: This vector is typically used as the user query in a Two-Tower Model for efficient candidate retrieval via Approximate Nearest Neighbor (ANN) search.

The self-supervised training objective where the model learns to predict the immediate next interaction given a user's preceding action history. This serves as a powerful proxy task for learning intent without explicit labels.

• Training Data: User sessions are treated as sequences of item IDs, with the model trained to maximize the probability of the true next item.
• Negative Sampling: To avoid computing a softmax over millions of items, techniques like Sampled Softmax or In-Batch Negatives are used, where other items in the mini-batch serve as negative examples.
• Output: The final hidden state corresponding to the last action is used as the user's current intent embedding for scoring the entire catalog.

The transformer's ability to directly connect a current action to a distantly past interaction, overcoming the vanishing gradient problem that plagues RNNs and LSTMs. This is crucial for capturing persistent, latent preferences.

• Example: A user who researched high-end cameras six months ago and now searches for a tripod. The transformer can directly attend to the camera research session, inferring the need for a professional-grade tripod.
• Mechanism: The attention weight matrix provides a direct, interpretable path showing which historical actions most influenced the current prediction.
• Contrast with RNNs: Recurrent models compress all history into a single fixed-size hidden state, losing fidelity over long sequences.

A technique that decomposes the final sequence representation into multiple distinct prototype vectors, each capturing a different latent interest. This addresses the limitation of a single embedding representing a multi-faceted user.

• Capsule Networks: Dynamic routing mechanisms can be applied to the transformer's output to cluster action representations into interest capsules.
• Diverse Recommendations: Instead of retrieving items similar to a single blended vector, the system queries the ANN index with each interest vector separately, ensuring coverage of all user facets (e.g., both 'workout gear' and 'formal wear').
• Interpretability: Each interest vector can often be visualized and labeled by the items it retrieves, providing explainability into the user's inferred preference clusters.