Behavior Sequence Transformer (BST)

Unlike RNNs that process sequences step-by-step, BST applies multi-head self-attention across the entire user interaction history in parallel. For each item in a sequence (e.g., clicked product, watched video), the mechanism computes attention weights against every other item. This allows the model to directly capture long-range dependencies—a click from 50 steps ago can directly influence the current prediction if semantically relevant, without information decaying through recurrent steps.

Since the Transformer has no inherent sense of order, BST injects sinusoidal positional encodings into the item embeddings. This adds a unique signal for each position t in the sequence, allowing the self-attention layers to distinguish between a user who clicked Item A then Item B versus Item B then Item A. This temporal grounding is critical for modeling sequential intent evolution.

BST's input layer fuses heterogeneous features into a unified representation:

• Item Features: Embeddings of the item ID, category, brand.
• Contextual Features: Time since last click, device type, day of week.
• Candidate Item: The target item being scored is concatenated with the sequence, allowing attention to focus on history relevant to this specific candidate. This dense feature fusion is what elevates BST beyond simple item-to-item sequence models.

A key operational advantage over LSTMs or GRUs is non-sequential computation. Because self-attention computes all pairwise interactions simultaneously, BST training can be massively parallelized across GPUs. This drastically reduces wall-clock training time on large-scale user behavior logs, making it feasible to train on sequences with hundreds of historical interactions.

The final architecture component is a multi-layer perceptron (MLP) sitting atop the pooled sequence representation. After the self-attention blocks process the user's history, the output is typically concatenated with other dense features and passed through fully connected layers with ReLU activations. The final sigmoid neuron outputs the predicted probability of a click, making BST a direct replacement for traditional deep CTR models.

The core computational unit enabling BST to model long-range dependencies without recurrence. For each item in a user's sequence, self-attention computes a weighted sum of all other items, allowing the model to directly connect a click from the distant past to a current candidate product. This parallel computation captures contextual relevance—such as relating an initial search for 'running shoes' to a later view of 'moisture-wicking socks'—without suffering from the vanishing gradient problems of RNNs.

Since the Transformer has no inherent sense of order, BST injects sinusoidal or learned positional vectors into item embeddings before the attention layers. This preserves the chronological structure of the sequence, ensuring the model distinguishes between a user who viewed Item A then Item B versus the reverse. In e-commerce, this temporal signal is critical for detecting intent escalation—recognizing that viewing a product page after reading reviews signals higher purchase intent than the opposite order.

Before entering the Transformer blocks, each interacted item is mapped to a dense, low-dimensional vector via an embedding lookup table. This layer compresses high-cardinality item catalogs into a continuous space where semantically similar products cluster together. BST often concatenates contextual features—such as category, brand, or price bucket—into the item representation, enriching the raw ID with side information that helps the attention mechanism generalize to cold-start items.

BST employs multiple attention heads operating in parallel, each learning a distinct relevance pattern from the same sequence. One head might focus on category-level transitions (Electronics → Accessories), while another captures price-sensitivity signals (viewing only discounted items). The outputs are concatenated and projected, giving the model a richer, multi-faceted understanding of user behavior than a single attention function could provide.

Each Transformer block in BST concludes with a position-wise feed-forward network that applies non-linear transformations to the attention outputs. Crucially, residual connections wrap both the attention and feed-forward sub-layers, followed by layer normalization. This architecture enables stable training of deep stacks—allowing BST to capture complex, hierarchical patterns in clickstreams without degradation, a key advantage over shallower sequential models like GRUs.

The final layer of BST is a softmax classifier that scores every candidate item in the catalog against the final hidden state of the sequence. During training, the model is optimized to maximize the probability of the actual next-clicked item. At inference, the top-K scores become the recommendation set. This objective directly aligns with session-based recommendation tasks, where the goal is to predict immediate intent from the current behavioral trajectory.