Self-Supervised Learning

A pre-training objective that randomly masks items in a user's interaction sequence and trains the model to predict the hidden items based on surrounding context.

• Mechanism: Analogous to BERT's masked language modeling, but applied to user action sequences
• Training Signal: The model learns co-occurrence patterns and sequential dependencies by reconstructing corrupted sequences
• Architecture: Typically uses bidirectional transformer encoders that attend to both past and future context within a session
• Example: Masking 15% of products in a browsing session and predicting them from the remaining 85%

A sequential training objective where the model predicts the immediate next interaction given a user's preceding action history, serving as a self-supervised proxy task for learning intent embeddings.

• Causal Masking: Uses unidirectional attention to ensure predictions only depend on past actions, preserving temporal causality
• Loss Function: Typically employs cross-entropy loss over the full item catalog or sampled softmax for efficiency
• Application: Forms the foundation of session-based recommenders and real-time personalization engines
• Example: Given a sequence [phone, case, screen protector], predicting 'wireless charger' as the next likely purchase

A representation learning paradigm that pulls semantically similar pairs closer in embedding space while pushing apart dissimilar pairs, often using the InfoNCE loss to define positive and negative samples.

• Positive Pairs: Derived from data augmentations of the same user session, such as cropping, masking, or reordering
• Negative Pairs: Randomly sampled from other users or sessions within the same batch
• Temperature Parameter: Controls the concentration of the distribution, sharpening similarity scores to focus on hard negatives
• Example: Two augmented views of the same browsing session should have high cosine similarity, while views from different users should be orthogonal

A metric learning objective that minimizes the distance between an anchor and a positive sample while maximizing the distance to a negative sample by a specified margin, enforcing relative similarity constraints.

• Triplet Structure: (anchor, positive, negative) where anchor and positive share semantic meaning
• Margin Parameter: Defines the minimum separation between positive and negative pair distances
• Hard Negative Mining: Selects negatives that are close to the anchor but incorrect, providing the strongest training signal
• Example: Anchor = user's current session, Positive = similar historical session, Negative = dissimilar user's session

A training efficiency technique that reuses other positive items within the same mini-batch as negative samples for a given query, leveraging the random composition of batches to approximate the full softmax distribution.

• Efficiency: Eliminates the need to compute scores against the entire item catalog, reducing computational cost by orders of magnitude
• Bias Correction: Requires correction for popularity bias, as frequently occurring items are more likely to appear as negatives
• Batch Size Dependency: Larger batches provide better approximations of the true distribution but increase memory requirements
• Example: In a batch of 1024 user-item pairs, each item serves as a positive for its paired user and a negative for all other 1023 users

A data augmentation strategy that creates positive pairs for contrastive learning by applying stochastic transformations to user behavior sequences while preserving the underlying intent signal.

• Crop: Randomly truncates a subsequence to simulate partial session observations
• Mask: Randomly drops items to simulate incomplete behavioral data
• Reorder: Shuffles a small local window to teach invariance to minor order variations
• Example: A session [A, B, C, D, E] might be augmented to [A, C, D] (crop + mask) and treated as a positive pair with the original

A self-supervised paradigm that learns representations by pulling semantically similar pairs together and pushing dissimilar pairs apart in embedding space. In user modeling, this means bringing session embeddings from the same user closer while separating embeddings from different users.

• Uses InfoNCE loss to define positive and negative pairs
• Positive pairs: augmented views of the same data instance
• Negative pairs: randomly sampled instances from the batch
• Critical for learning robust user embeddings without explicit labels

A pre-training objective inspired by BERT that randomly masks items in a user's interaction sequence and trains the model to predict the hidden items from surrounding context. This forces the model to learn deep bidirectional representations of user intent.

• Masks 15-30% of items in a behavioral sequence
• Model must reconstruct masked items from left and right context
• Produces sequence-aware user embeddings that capture temporal dependencies
• Enables zero-shot transfer to downstream personalization tasks

A sequential self-supervised training objective where the model learns to predict the immediate next interaction given a user's preceding action history. This proxy task teaches the model to capture evolving user intent and short-term session dynamics.

• Training signal derived from natural temporal ordering of actions
• No manual labeling required—the next click is the label
• Produces embeddings that encode transitional patterns between items
• Foundation for session-based recommendation systems

A metric learning objective that structures the embedding space by enforcing relative distance constraints. For user embeddings, an anchor session is pulled closer to a positive session from the same user and pushed away from a negative session from a different user by a specified margin.

• Anchor: reference data point
• Positive: semantically similar instance
• Negative: semantically dissimilar instance
• Margin hyperparameter controls separation strength
• Produces well-clustered user representations

An efficient training technique that reuses other positive examples within the same mini-batch as negatives for a given query. This eliminates the need for a separate negative sampling step and dramatically improves training throughput for large-scale user embedding models.

• Negatives are other users' items in the same batch
• Approximates full softmax over the entire item catalog
• Scales efficiently with batch size—larger batches provide more negatives
• Standard practice in two-tower retrieval model training

Techniques that create multiple valid views of the same behavioral sequence to generate positive pairs for contrastive learning. Augmentations preserve the underlying user intent while introducing controlled variation.

• Item masking: randomly hide items in the sequence
• Item cropping: extract contiguous subsequences
• Item reordering: shuffle small local windows
• Item substitution: replace items with semantically similar alternatives
• Critical for learning invariant user representations