Self-Attentive Sequential Recommendation (SASRec)

SASRec applies a causal self-attention mask to the user's item interaction sequence. Each item embedding attends only to preceding items, preserving temporal order. This mechanism computes a weighted sum of past interactions, where the attention weights are dynamically learned based on the relevance of each historical item to the current prediction context.

• Key distinction: Unlike bidirectional attention in BERT4Rec, SASRec's unidirectional design mirrors the natural left-to-right flow of user behavior.
• Scaled dot-product attention is used: Attention(Q, K, V) = softmax(QK^T / √d)V, with a lower-triangular mask preventing future information leakage.

Since self-attention is inherently permutation-invariant, SASRec injects learnable positional embeddings into the item representation at the input layer. These embeddings are trained end-to-end, allowing the model to discover optimal position representations for the specific recommendation domain.

• Implementation: A positional embedding matrix P ∈ R^N×d is learned, where N is the maximum sequence length and d is the hidden dimensionality.
• Input construction: The input to the first block is E_item + P_position, summing the item embedding and its corresponding positional embedding.
• This contrasts with the fixed sinusoidal encodings used in the original Transformer, offering greater flexibility for sequential recommendation tasks.

SASRec stacks multiple Transformer encoder-style blocks to learn increasingly complex item transition patterns. Each block contains a multi-head self-attention sub-layer followed by a position-wise feed-forward network.

• Residual connections wrap each sub-layer: LayerNorm(x + Sublayer(x)), enabling stable gradient flow through deep architectures.
• Dropout regularization is applied to attention weights and feed-forward activations to prevent overfitting on sparse interaction data.
• Layer normalization is applied before each sub-layer (pre-norm variant), which empirically improves training stability for recommendation tasks.

Each self-attention block includes a position-wise feed-forward network (FFN) applied identically to each sequence position. This introduces non-linearity and increases model capacity independently of the attention mechanism.

• Structure: A two-layer MLP with ReLU activation: FFN(x) = ReLU(xW₁ + b₁)W₂ + b₂.
• Dimensionality: The inner layer typically expands to 4× the hidden dimension before projecting back, following the original Transformer design.
• The FFN allows the model to learn item-specific transformations that complement the relational patterns captured by attention.

SASRec is trained using binary cross-entropy loss on implicit feedback data. For each positive user-item interaction in the sequence, one or more negative items are randomly sampled from the catalog.

• Loss formulation: -Σ[log(σ(r_pos)) + Σ log(1 - σ(r_neg))], where σ is the sigmoid function and r is the predicted relevance score.
• Shared item embedding matrix: The same item embeddings are used in both the input (as sequence items) and output (as prediction targets), reducing parameters and enforcing consistency.
• This pairwise approach treats recommendation as a binary classification task, distinguishing observed interactions from unobserved ones.

At inference time, SASRec takes the hidden representation of the final item in the input sequence and computes its dot product with all candidate item embeddings to produce ranking scores.

• Scoring: score_i = h_t · E_i, where h_t is the final hidden state and E_i is the embedding of candidate item i.
• Efficiency: This enables pre-computation of item embeddings and fast top-K retrieval using Approximate Nearest Neighbor (ANN) libraries like FAISS or ScaNN.
• The model naturally handles variable-length sequences, adapting to users with both short and extensive interaction histories.

The broader paradigm that SASRec instantiates, focused on capturing temporal dependencies in user action sequences. Unlike session-based approaches that treat interactions as unordered sets, sequential modeling preserves the chronological order of clicks, purchases, and views to predict next-item intent. Key distinctions include:

• Short-term sessions vs. long-term lifecycles
• Unidirectional (causal) vs. bidirectional context windows
• Point-of-interest and next-basket recommendation variants

A self-supervised pre-training strategy often paired with SASRec-style encoders to improve representation quality. The model learns by pulling together embeddings of augmented views of the same sequence while pushing apart embeddings from different sequences. Common augmentations include:

• Item masking: randomly hiding items in the input sequence
• Item cropping: truncating subsequences
• Item reordering: shuffling small contiguous windows This pre-training produces more robust sequence encoders before fine-tuning on the next-item prediction objective.

A critical training efficiency technique that makes SASRec tractable on massive item catalogs. Instead of computing the full softmax over millions of items, the loss is computed using only the positive target item and a small set of randomly sampled negative items. The original SASRec paper uses a binary cross-entropy loss with one positive and one randomly sampled negative per time step. Modern variants employ:

• In-batch negatives: reusing other sequences' targets as negatives
• Hard negative mining: sampling items with high model scores that are not the true target
• Popularity-based sampling: correcting for item frequency bias

The standard evaluation metrics used to benchmark SASRec against baselines like GRU4Rec and Caser. Recall@K measures the proportion of ground-truth next items that appear in the top-K predictions, directly assessing retrieval coverage. Normalized Discounted Cumulative Gain (NDCG@K) weights hits by their rank position, rewarding models that place the correct item higher in the list. Typical evaluation protocol:

• Leave-one-out: hold out the last interaction per user for testing
• Sampled metrics: rank the true item against a random subset of negatives
• Full-ranking: evaluate against the entire catalog for production-realistic assessment

A model compression technique where a smaller student SASRec model is trained to mimic the output distribution of a larger, more accurate teacher SASRec model. The student minimizes a combined loss of:

• Hard-label loss: standard next-item prediction cross-entropy
• Soft-label loss: KL divergence between student and teacher output logits This enables deployment of self-attentive recommenders in low-latency production environments where the full teacher model's inference cost would violate serving SLAs, while retaining most of the ranking quality.

The fundamental challenge SASRec faces with new users who have no interaction history to form a sequence. Without past items, the self-attention mechanism has no keys or values to attend over. Mitigation strategies include:

• Content-based fallback: using item metadata embeddings when behavioral signals are absent
• Demographic priors: initializing user representations from profile features
• Exploration policies: employing contextual bandits to gather initial interactions
• Zero-shot transfer: pre-training on auxiliary user populations and fine-tuning with few shots