Factorization Machine (FM)

FMs excel where traditional models fail: high-dimensional, sparse feature spaces. By factorizing interactions into low-rank latent vectors, FMs estimate parameters for feature combinations that never co-occurred in the training data.

• Mechanism: Each feature learns a dense embedding vector; interactions are modeled as the dot product of these vectors.
• Example: Predicting a user's affinity for a new artist based on their affinity for similar genres, even if the specific user-artist pair is unseen.
• Advantage: No manual feature engineering for cross-features is required.

Despite modeling all pairwise feature interactions, FMs can be computed in O(kn) time, where k is the factorization dimension and n is the number of non-zero features.

• Mathematical Reformulation: The pairwise interaction term is rewritten using the property: sum over all pairs of dot products equals a function of the squared sum of embeddings.
• Result: Training and prediction scale linearly with the number of features, not quadratically.
• Impact: FMs handle datasets with millions of categorical features without computational explosion.

FMs are a unified model that subsumes many specialized algorithms by controlling the input feature representation.

• Regression: Standard FM with real-valued target.
• Binary Classification: FM with a logit or probit link function.
• Ranking: FM optimized with pairwise loss functions like Bayesian Personalized Ranking (BPR).
• Matrix Factorization: FM with only user and item indicator variables as input exactly mimics standard MF.
• SVD++ and PITF: Achieved by adding additional context features (e.g., time, implicit history) to the input.

FMs naturally incorporate side information beyond user-item IDs, solving the cold-start problem inherent in pure collaborative filtering.

• Context Features: User demographics (age, location), item attributes (category, price), and temporal signals (time of day, day of week) are added as additional one-hot encoded features.
• Shared Embeddings: The latent vector for a feature like 'genre=rock' is shared across all interactions, transferring knowledge between users and items.
• Example: A new item with only metadata can still receive accurate predictions because its attribute embeddings are already trained from other items.

The standard FM models second-order (pairwise) interactions. Higher-Order Factorization Machines (HOFMs) extend this to capture complex multi-feature relationships.

• d-way Interactions: Model interactions among 3 or more features using higher-order tensor factorization.
• Field-aware FM (FFM): A variant that learns multiple latent vectors per feature, one for each interacting field (e.g., user field vs. item field), capturing field-specific interaction strengths.
• Trade-off: Increased expressiveness comes at higher computational cost; FFMs are O(kn²) in practice.

FMs serve as a powerful shallow component in hybrid deep learning architectures, combining memorization with generalization.

• DeepFM: A neural network that jointly trains an FM component for low-order feature interactions and a deep neural network for high-order non-linear patterns, sharing the same input embedding layer.
• Neural Factorization Machine (NFM): Replaces the FM's simple dot product with a neural network applied to the element-wise product of feature embeddings.
• xDeepFM: Introduces a Compressed Interaction Network (CIN) to learn vector-wise feature interactions at a bit-wise level, complementing the implicit high-order interactions of a DNN.

The process of creating synthetic features by combining two or more individual features, often via a Hadamard product or concatenation, to model non-linear interactions and co-occurrence patterns between them. Factorization Machines automate this by learning a latent vector for each feature, making the interaction strength between any two features the dot product of their embeddings. This eliminates the need for manual feature engineering and allows the model to estimate interactions even for pairs never observed together in the training data.

A latent factor model that decomposes the sparse user-item interaction matrix into a product of two dense, lower-dimensional matrices representing user and item latent vectors. Factorization Machines generalize this concept: while standard Matrix Factorization only models user-item interactions, FMs can incorporate arbitrary side features (time, context, item metadata) and model interactions between any feature types. In the special case with only user and item IDs, an FM reduces to standard Matrix Factorization.

A trainable lookup table that maps high-cardinality categorical features into dense, low-dimensional continuous vector representations. In Factorization Machines, each feature value is assigned a k-dimensional latent vector (the embedding). The model's core innovation is that the interaction between two features is modeled as the inner product of their embeddings, allowing the model to share statistical strength across sparse features and generalize to unseen combinations.

A recommendation methodology that makes automatic predictions about a user's interests by collecting preferences from many users. Factorization Machines serve as a general-purpose collaborative filtering engine that extends beyond user-item matrices. By encoding user demographics, item attributes, and contextual signals as features, FMs perform collaborative filtering with rich side information, addressing the cold start problem more effectively than pure matrix factorization approaches.

The task of estimating the probability that a user will click on a specific item or advertisement. Factorization Machines are widely used in CTR prediction because they efficiently handle the extremely sparse, high-dimensional categorical feature spaces common in advertising data. The model captures pairwise feature interactions—such as the relationship between a user's age group and an ad's category—without the combinatorial explosion of explicit feature crossing, making it suitable for real-time bidding environments.

A linear model extended with polynomial feature combinations to capture non-linear relationships. A second-degree Factorization Machine can be viewed as a factorized polynomial regression where the weight of each interaction term is not an independent parameter but the inner product of two latent vectors. This factorization drastically reduces the number of parameters from O(n²) to O(nk), where k is the embedding dimension, preventing overfitting on sparse data while still modeling all pairwise interactions.