Matrix Factorization

The core mechanism involves learning a dense embedding vector for every user and every item in a shared, low-dimensional latent space. Each dimension represents a hidden, unobserved attribute or taste factor. A user's affinity for an item is predicted by the dot product or cosine similarity between their respective vectors, mapping both entities into a space where proximity implies preference.

ALS is the dominant algorithm for large-scale implicit feedback datasets. It works by iteratively optimizing one set of latent factors while keeping the other fixed:

• Fix item vectors: Solve for all user vectors in parallel.
• Fix user vectors: Solve for all item vectors in parallel. This alternating non-convex optimization is massively parallelizable, making it ideal for distributed computing on billion-scale matrices where Stochastic Gradient Descent (SGD) would be inefficient.

The model adapts to different signal types through its objective function:

• Explicit Feedback: Predicts a numerical rating (e.g., 1-5 stars) by minimizing a regularized squared error loss, often ignoring missing entries.
• Implicit Feedback: Treats interactions like clicks, views, or purchases as a confidence-weighted binary signal. The model learns to predict a preference score, assigning higher confidence to observed interactions but still learning from the absence of an interaction as a low-confidence negative signal.

To ensure the model generalizes to unseen user-item pairs rather than memorizing the sparse input, L2 regularization is applied to the latent vectors. This penalizes large vector magnitudes, effectively keeping the latent space smooth and preventing the model from fitting noise. The regularization parameter (lambda) is a critical hyperparameter tuned via cross-validation to balance bias and variance.

While pure collaborative filtering struggles with new entities, matrix factorization can be extended to incorporate side features to alleviate the cold start problem:

• User features: Demographics, device type, or sign-up source can be concatenated or summed with the learned user embedding.
• Item features: Brand, category, price, or content embeddings from a separate model can augment the item vector. This hybrid approach allows the model to make informed predictions even without any interaction history.

For production retrieval, the factorization process separates neatly into two stages:

• Offline Training: The ALS algorithm computes the final user and item latent factor matrices.
• Online Serving: Item vectors are indexed in an Approximate Nearest Neighbor (ANN) service like FAISS or ScaNN. A user's vector is used as a query to retrieve the top-K items with the highest dot product in milliseconds, decoupling the model from the real-time retrieval infrastructure.

The broader recommendation paradigm that matrix factorization operationalizes. Collaborative filtering predicts user preferences by aggregating behavioral patterns across many users, operating on the principle that users who agreed in the past will agree in the future.

• Memory-based CF: Computes user-user or item-item similarity directly from the interaction matrix
• Model-based CF: Learns latent representations, with matrix factorization being the canonical approach
• Implicit feedback variants: Weighted alternating least squares handles confidence levels in observed interactions rather than explicit ratings

The neural implementation of latent factor vectors. An embedding layer is a trainable lookup table that maps high-cardinality categorical features—such as user IDs or item IDs—into dense, low-dimensional continuous vectors.

• Each row corresponds to a user or item's latent representation
• During training, backpropagation updates these embeddings just as ALS updates factor matrices
• In two-tower architectures, embedding layers feed into separate user and item towers for efficient retrieval
• Dimensionality typically ranges from 32 to 256, balancing expressiveness with computational cost

A pairwise learning-to-rank optimization criterion that directly addresses the implicit feedback setting. Rather than predicting exact ratings, BPR maximizes the probability that a user prefers an observed item over an unobserved item.

• Treats recommendation as a binary classification on item triplets: (user, positive item, negative item)
• Loss function: maximizes the margin between positive and negative item scores
• Pairs naturally with matrix factorization, replacing the standard MSE loss
• Provides superior top-N ranking performance compared to pointwise regression on implicit data

A framework that generalizes matrix factorization by replacing the inner product with a neural architecture. While standard MF models user-item interaction as a linear dot product, NCF learns arbitrary non-linear interaction functions.

• GMF layer: Element-wise product of user and item embeddings, equivalent to classical MF
• MLP layer: Concatenates embeddings and passes them through deep fully-connected layers
• NeuMF: Fuses both pathways for hybrid linear and non-linear modeling
• Demonstrates that MF is a special case within a broader neural recommendation paradigm

A general-purpose supervised learning algorithm that models pairwise feature interactions as the dot product of latent factor vectors. FMs extend matrix factorization beyond user-item matrices to arbitrary feature sets.

• Handles sparse categorical features where interactions may never co-occur in training data
• Each feature receives a latent vector; interaction strength is computed via dot product
• Field-aware FM (FFM) extends this by learning field-specific latent vectors
• Widely used in CTR prediction before deep learning became dominant
• Computational trick: reformulates pairwise interactions to O(kn) instead of O(kn²)

The fundamental limitation of pure collaborative matrix factorization. When a new user or new item enters the system with no historical interactions, their latent vector cannot be reliably estimated from the interaction matrix alone.

• User cold start: No behavioral history to infer preferences
• Item cold start: No interaction data to determine which users would engage
• Mitigation strategies include:
  • Using content features or metadata to initialize embeddings
  • Hybrid models combining collaborative and content-based signals
  • Active exploration policies to gather initial feedback
• This limitation directly motivates deep learning architectures that ingest side information