Factorization Machines (FM)

FM models all pairwise feature interactions using factorized latent vectors, achieving the expressiveness of polynomial regression while maintaining linear computational complexity O(kn). Instead of learning a separate weight for every feature pair, FM learns a low-rank embedding for each feature, drastically reducing parameters from O(n²) to O(kn).

FM estimates interaction parameters reliably even when feature combinations never co-occur in training data. By factorizing the interaction matrix into latent vectors, the model learns through transitive relationships: if feature A interacts with B, and B interacts with C, FM can infer the A-C interaction without direct observation. This is critical for recommender systems where user-item matrices are over 99% empty.

FM accepts a single real-valued feature vector, making it a universal predictor that handles diverse input types through smart encoding:

• Categorical variables: one-hot encoded into sparse binary indicators
• Numerical variables: passed directly as real values
• Metadata context: user age, time of day, device type This eliminates the need for separate collaborative filtering and content-based models.

Field-aware Factorization Machines extend FM by learning multiple latent vectors per feature—one for each feature field it interacts with. A feature representing a user's age learns different embeddings when interacting with item price versus item category. This captures field-specific interaction semantics at the cost of increased parameters, making FFM the winning model in multiple CTR prediction competitions.

FM serves as a foundational building block in modern multi-task learning architectures. Its factorized embeddings can be shared across related objectives—predicting clicks, conversions, and ratings simultaneously. The shared latent representations act as a regularization mechanism, improving generalization on tasks with limited data while maintaining task-specific output layers.

FM inherently mitigates the cold start problem through its factorization mechanism. New items or users with minimal interaction history still receive meaningful predictions because their latent vectors are estimated from side features (category, brand, demographics) rather than relying solely on collaborative signals. This enables immediate personalization without waiting for behavioral data accumulation.

An extension of FM that learns multiple latent vectors for each feature—one for every other feature field. While standard FM uses a single vector v_i to model interactions, FFM uses v_{i,F(j)} to capture the nuanced relationship between a feature and the specific field it interacts with.

• Field concept: Categorical groupings like 'User', 'Item', 'Context'
• Trade-off: Higher accuracy at the cost of a quadratic increase in parameters
• Winning model: Dominated CTR prediction competitions (Criteo, Avazu) before deep learning

The foundational concept that FM automates. Manual feature crossing requires domain experts to explicitly create interaction terms like user_age × item_price. FM learns these implicitly through latent vector dot products.

• Polynomial regression requires O(n²) parameters for degree-2 interactions
• FM reduces this to O(nk) where k is the latent dimension
• Enables modeling interactions between features that never co-occur in training data

The primary reason FM outperforms polynomial regression on recommendation data. In a user-item interaction matrix with millions of IDs, most feature combinations appear zero times.

• Collaborative filtering: FM generalizes matrix factorization by incorporating side features
• Example: A user who clicked 'sci-fi books' and 'space documentaries' generates a latent user vector that interacts strongly with 'NASA merch' even without direct history
• Cold start mitigation: New items with metadata but no interactions still receive meaningful latent vectors

The standard objective function for training FM on CTR prediction tasks. Also called binary cross-entropy, it measures the divergence between predicted click probability and the true binary label.

• Formula: - [y log(p) + (1-y) log(1-p)]
• Penalty: Heavily punishes confident wrong predictions (predicting 0.99 for a non-click)
• Gradient: Drives the factorization via stochastic gradient descent (SGD) or Adam
• Calibration: Well-optimized log loss produces well-calibrated probability estimates

Google's architecture that combines a wide linear model for memorizing specific feature co-occurrences with a deep neural network for generalization. The wide component essentially performs what FM does—capturing feature interactions—while the deep part learns higher-order patterns.

• Joint training: Both components are trained simultaneously on the same loss
• FM as wide component: Some implementations replace the wide part with an FM layer
• Evolution: Led to DeepFM, which combines FM and DNN without manual feature engineering

The neural network equivalent of FM's latent vectors. An embedding layer maps sparse categorical features (like user ID 582913) into dense, low-dimensional vectors that capture semantic similarity.

• FM connection: The latent vectors v_i are exactly the embedding weights
• Lookup operation: During inference, the embedding for feature i is retrieved via a simple table lookup
• End-to-end training: Embeddings are learned jointly with the prediction task, making them task-specific representations