Embedding Layer

The primary function is to map high-cardinality categorical features (e.g., millions of item IDs) from a massive, sparse one-hot encoded space into a dense, low-dimensional vector (e.g., 32–256 dimensions).

• One-Hot Input: A 1,000,000-dimensional vector with a single '1'.
• Dense Output: A compact 64-dimensional vector of floating-point numbers.
• Memory Efficiency: Instead of storing a massive sparse matrix, the layer stores a lookup table of size vocabulary_size × embedding_dim.

Through backpropagation, the model learns to position vectors such that items with similar user interaction patterns are close in the vector space. This captures collaborative filtering signals without explicit feature engineering.

• Cosine Similarity: The primary metric for measuring vector closeness.
• Analogy Property: The vector arithmetic vector('King') - vector('Man') + vector('Woman') ≈ vector('Queen') applies to items; vector('Item A') - vector('Category X') + vector('Category Y') finds analogous items in the new category.
• Visualization: t-SNE or UMAP projections often reveal natural item clusters based on genre, brand, or use case.

An embedding layer is functionally a trainable weight matrix of shape (vocabulary_size, embedding_dim). During a forward pass, it performs an efficient integer-based lookup, not a full matrix multiplication.

• Gradient Flow: Gradients flow back only to the row corresponding to the active input ID, updating only that specific entity's vector.
• Out-of-Vocabulary Handling: A dedicated 'unknown' index is often added to the vocabulary to catch unseen IDs during inference.
• Initialization: Weights are typically initialized with a small random uniform or normal distribution to break symmetry before learning.

When a user has interacted with a sequence of items, their IDs form a multi-hot vector. The embedding layer maps each ID to a vector, and a pooling operation combines them into a single fixed-length user representation.

• Average Pooling: Computes the element-wise mean of all item vectors in the sequence.
• Sum Pooling: Computes the element-wise sum, preserving the magnitude of the interaction count.
• Attention Pooling: Used in architectures like Deep Interest Network (DIN), where a candidate item's embedding acts as a query to compute a weighted sum of historical item vectors, focusing on the most relevant past behaviors.

In Multi-Task Learning (MTL) architectures, a key design choice is whether to share embedding tables across tasks or maintain separate ones.

• Shared Embeddings: All tasks (e.g., CTR and CVR prediction) use the same underlying item vectors. This acts as a strong regularizer and transfers knowledge, but can suffer from negative transfer if tasks are too dissimilar.
• Task-Specific Embeddings: Each task learns its own embedding space. This maximizes flexibility but increases memory footprint and prevents cross-task generalization.
• Hybrid Approach: Architectures like MMoE share a base embedding layer but route the output through task-specific gating networks and expert subnetworks.

For features with an unbounded or extremely large cardinality (e.g., raw search queries), a standard embedding table is infeasible. The hashing trick maps an arbitrary string to a fixed-size bucket using a hash function, and an embedding is learned for each bucket.

• Collision Tolerance: Multiple distinct features will map to the same bucket, introducing noise. The model learns to be robust to these collisions, treating the hashed feature as a statistical aggregation.
• Fixed Memory Footprint: The vocabulary size is capped at the number of hash buckets, guaranteeing a constant memory allocation regardless of data scale.
• No Vocabulary Building: Eliminates the need to pre-compute a vocabulary, simplifying online learning pipelines.

The preprocessing step that converts a categorical feature into a sparse binary vector before it enters the embedding layer.

• A vocabulary of 1 million item IDs creates a 1-million-dimension vector with a single '1'.
• The embedding layer multiplies this sparse vector by a weight matrix to produce a dense representation.
• Without one-hot encoding, categorical features cannot be mapped to trainable embeddings.

A dimensionality reduction technique that maps high-cardinality categorical features into a fixed-size vector using a hash function, bypassing the need for a full vocabulary.

• The hashing trick allows embedding layers to handle unseen categories without a pre-built dictionary.
• Collisions occur when two different features hash to the same bucket, introducing noise.
• Commonly used when feature cardinality is unknown or exceeds memory constraints.

A supervised algorithm that models pairwise feature interactions using factorized latent vectors, effectively learning embeddings for each feature.

• Each feature receives a k-dimensional latent vector analogous to an embedding.
• The dot product between two latent vectors captures the interaction strength.
• FMs serve as a bridge between matrix factorization and deep neural networks in CTR prediction.

An attention-based architecture where the embedding layer produces behavioral embeddings that are adaptively weighted against a candidate item.

• User behavior sequences are embedded and passed through an attention mechanism.
• The model learns which historical actions are relevant to the current item.
• Embedding quality directly determines the effectiveness of the attention weighting.

A joint architecture where the deep component relies on embedding layers to generalize to unseen feature combinations.

• Categorical features are embedded into dense vectors for the deep network.
• The wide component memorizes sparse feature crosses using one-hot encoded inputs.
• Embedding dimensions are a critical hyperparameter balancing memorization and generalization.

A multi-task architecture where shared embedding layers feed multiple expert subnetworks, each gated differently per task.

• A single user or item embedding serves as input to all experts.
• Task-specific gating networks learn which experts to activate for CTR vs. CVR prediction.
• Shared embeddings enable transfer learning across related objectives.