Dimensionality Reduction

A linear transformation that identifies the orthogonal axes of maximum variance in the data. PCA projects embeddings onto the top principal components, discarding dimensions with minimal information. In user embedding systems, PCA is often applied to embedding tables post-training to reduce the embedding dimension from 256 to 64, cutting storage by 75% while preserving the dominant preference signals. The computation relies on eigendecomposition of the covariance matrix, making it deterministic and fast but limited to capturing linear relationships.

A non-linear technique designed specifically for visualization. t-SNE converts high-dimensional Euclidean distances into conditional probabilities representing similarities, then minimizes the Kullback-Leibler divergence between the high-dimensional and low-dimensional distributions. The heavy-tailed Student-t distribution in the low-dimensional space prevents the crowding problem, allowing distinct clusters to separate. It is the standard tool for qualitatively inspecting user embedding clusters to validate that users with similar browsing patterns group together.

A manifold learning technique that preserves both local and global structure better than t-SNE. UMAP constructs a fuzzy topological representation of the high-dimensional data using nearest-neighbor graph theory, then optimizes a low-dimensional embedding via cross-entropy minimization. It scales significantly faster than t-SNE and supports transforming new data points without full retraining. In production, UMAP is used to project sequence-aware embeddings into 2D or 3D for real-time dashboards monitoring embedding drift.

A neural network approach where an encoder compresses the input embedding into a low-dimensional latent bottleneck, and a decoder reconstructs the original vector. The reconstruction loss forces the bottleneck to capture the essential manifold structure. Unlike PCA, autoencoders with non-linear activations can learn complex, non-linear compression schemes. This technique is particularly effective for hybrid embeddings that fuse collaborative and content signals, as the autoencoder learns to preserve the interaction patterns that matter most for cosine similarity scoring.

A computationally lightweight method based on the Johnson-Lindenstrauss lemma, which states that high-dimensional points can be projected into a much lower-dimensional space while approximately preserving pairwise distances. A random matrix with entries drawn from a Gaussian or sparse distribution multiplies the original embedding. Despite its simplicity, random projection is highly effective for locality-sensitive hashing and serves as a fast pre-processing step before approximate nearest neighbor indexing, reducing the index build time for billion-scale embedding tables.

The mathematical foundation of PCA, SVD factorizes the user-item interaction matrix into three constituent matrices. By keeping only the top-k singular values and corresponding vectors, a low-rank approximation is achieved. This is the core mechanism behind collaborative filtering embedding generation. Truncated SVD directly produces compact user and item vectors in a shared latent space where dot products reconstruct the original interaction matrix. It remains a strong baseline for cold-start embedding scenarios where only sparse interaction data is available.