Embedding Table

An embedding table is a 2D matrix of shape [vocabulary_size, embedding_dimension]. Each row corresponds to a unique categorical value (e.g., a user_id or item_sku), and each column represents a latent feature dimension. During a forward pass, the input sparse index is used to perform a direct row lookup, retrieving the corresponding dense vector. This operation is highly parallelizable on GPUs, making it the standard for scaling deep learning recommenders to billions of categorical features.

The core operation is an EmbeddingBag or gather function, which maps a list of integer indices to their corresponding dense vectors. Frameworks like PyTorch and TensorFlow implement this as a fused CUDA kernel that combines the memory lookup with a reduction operation (sum, mean) in a single pass. This avoids materializing large intermediate tensors, keeping the memory bandwidth bottleneck manageable. For distributed training, tables are often sharded across multiple GPUs using model parallelism to handle vocabularies exceeding the memory of a single accelerator.

Mapping raw categorical strings to integer indices requires a vocabulary. For high-cardinality features with long-tailed distributions, feature hashing is often used to map values to a fixed-size table without maintaining an explicit mapping. The hashing trick applies a deterministic hash function (e.g., MurmurHash) to the raw feature value, modulo the table size. This trades the risk of hash collisions for a bounded memory footprint and eliminates the need to synchronize a vocabulary dictionary across distributed workers.

When a single example contains multiple categorical features that map to the same embedding table (e.g., a user's last 10 viewed items), the individual vectors must be combined into a single fixed-length representation. Common reduction modes include:

• Sum: Adds the vectors element-wise, preserving magnitude information.
• Mean: Averages the vectors, normalizing for sequence length variance.
• Concat: Concatenates vectors, preserving individual identity at the cost of dimensionality. The choice of pooling impacts the inductive bias of the downstream network.

Proper weight initialization is critical for convergence. Embeddings are typically initialized with a uniform distribution in the range [-sqrt(3/dim), sqrt(3/dim)] or a normal distribution with a small standard deviation. To combat overfitting on rare feature values, L2 regularization is applied directly to the embedding vectors, pulling them toward the origin. Dropout can also be applied to the pooled embedding output, randomly zeroing entire dimensions to prevent co-adaptation.

For web-scale models with embedding tables exceeding terabytes, storing the full table in GPU HBM is impossible. Architectures employ a parameter server or SSD-backed cache where the master copy of the table resides in CPU RAM or NVMe storage. During training, only the mini-batch's active rows are fetched to the GPU. This hybrid storage hierarchy uses a least-recently-used (LRU) eviction policy to keep hot embeddings on the accelerator, balancing the cost of PCIe data transfer against the speed of local lookups.