Feature Crossing

Linear models learn independent weights for each feature, making them unable to represent relationships like 'users in New York buy coffee in the morning.' A feature crossing explicitly creates a new feature for the conjunction New York AND morning, allowing the model to learn a distinct weight for this specific co-occurrence. This extends linear models into highly expressive territories without the complexity of deep neural networks.

Feature crossing enables the memorization of rare but highly predictive feature pairs. If a specific user ID and item ID have a historical interaction, a crossing of these two features creates a sparse feature that acts as a direct lookup. This allows the model to perfectly memorize that this user clicks on this item, a capability that complements the generalization provided by embeddings in architectures like the Wide & Deep Learning model.

The primary engineering challenge of feature crossing is the combinatorial explosion of the feature space. Crossing a feature with 10,000 categories with another with 10,000 categories can theoretically produce 100 million new sparse features. In practice, this is managed by:

• Feature Hashing: Mapping the crossed feature to a fixed-size vector.
• Frequency Filtering: Discarding crossings that occur below a minimum threshold.
• Dimensionality Reduction: Using techniques like PCA on the crossed feature space.

Feature crossing is not limited to categorical variables. A numerical feature like time_of_day can be discretized into buckets (e.g., 'morning', 'afternoon', 'evening') and then crossed with a categorical feature like product_category. This creates features like electronics_x_evening, allowing the model to learn that electronics browsing peaks during evening hours. The discretization step is critical to avoid creating an infinite number of continuous crossings.

While both capture feature relationships, they serve different purposes. Embeddings learn dense, low-dimensional representations that generalize across similar entities. Feature crossings provide explicit, high-dimensional memorization of exact co-occurrences. Modern architectures like the Wide & Deep model combine both: the 'wide' component uses feature crossings for memorization, while the 'deep' component uses embeddings for generalization. This hybrid approach is a standard pattern in production CTR prediction systems.

Manually engineering feature crosses is labor-intensive and requires deep domain expertise. Factorization Machines (FM) automate this by modeling all pairwise feature interactions using factorized latent vectors. Instead of creating an explicit weight for every crossed pair, FM learns an embedding vector for each feature and models the interaction as the dot product of these vectors. This drastically reduces the parameter count from O(n²) to O(n*k), where k is the embedding dimension, making it feasible for high-cardinality datasets.

A supervised learning algorithm that models pairwise feature interactions using factorized latent vectors. Instead of learning a separate weight for every single feature combination, FM learns a low-dimensional embedding for each feature. The interaction between two features is then modeled as the dot product of their latent vectors. This is computationally efficient and allows the model to estimate interactions even for feature pairs that rarely or never co-occur in the training data, solving the sparsity problem that plagues standard polynomial regression.

An extension of Factorization Machines that captures the nuanced interaction between different categories of variables. While FM learns a single latent vector per feature, FFM learns multiple vectors—one for every other feature field. For example, a feature like 'ESPN' in the 'Publisher' field learns one latent vector to interact with the 'Advertiser' field and a different vector to interact with the 'User Gender' field. This field-awareness significantly increases model expressiveness but also raises complexity to O(kn²).

A dimensionality reduction technique that maps high-cardinality categorical features into a fixed-size vector using a hash function. When creating feature crosses, the combinatorial explosion of categories (e.g., 'city x language') can create billions of sparse features. Hashing maps these to a smaller, pre-defined space (e.g., 1 million buckets), controlling memory and model size. A key trade-off is hash collisions, where two different feature crosses map to the same bucket, introducing noise but often with negligible performance impact at scale.

The standard evaluation metric and loss function for CTR prediction models that use feature crosses. It quantifies the accuracy of a probabilistic classifier by heavily penalizing confident but incorrect predictions. For a true label y and a predicted probability p, the loss is -(y log(p) + (1-y) log(1-p)). A model that confidently predicts a 99% click probability for a non-clicked item incurs a massive penalty, driving the optimization of interaction weights during training.