Dimensionality Reduction

A linear technique that identifies orthogonal axes of maximum variance in the data. PCA computes the eigenvectors of the covariance matrix to project embeddings onto a lower-dimensional subspace. Key properties:

• Deterministic and computationally efficient
• Preserves global structure but may collapse local neighborhoods
• Often used as a baseline before applying non-linear methods
• Example: Reducing a 768-dim BERT embedding to 50 dimensions for clustering

A non-linear technique that converts pairwise Euclidean distances into conditional probabilities representing similarities. t-SNE minimizes the KL divergence between high-dimensional and low-dimensional probability distributions. Key properties:

• Exceptional at preserving local neighborhood structure
• Computationally expensive; scales poorly beyond ~10k points
• Stochastic output varies between runs
• Primarily used for visualization, not as a preprocessing step
• Perplexity hyperparameter critically controls the balance between local and global structure

A manifold learning technique based on Riemannian geometry and algebraic topology. UMAP constructs a fuzzy topological representation of the high-dimensional data and optimizes a low-dimensional embedding to be structurally similar. Key properties:

• Preserves both local and global structure better than t-SNE
• Significantly faster; scales to millions of points
• Supports supervised and semi-supervised dimensionality reduction
• Produces a learnable transform that can be applied to new data
• Parameters n_neighbors and min_dist control the granularity of structure preservation

Neural network architectures trained to reconstruct their input through a bottleneck layer. The encoder compresses the input into a low-dimensional latent code, and the decoder reconstructs the original data from that code. Key properties:

• Learns non-linear, data-specific compression functions
• Variational Autoencoders (VAEs) enforce a continuous, smooth latent space
• Can be stacked to create deep hierarchical representations
• Useful for denoising embeddings by learning to ignore irrelevant variance
• Example: Training a 3-layer autoencoder to compress 1024-dim embeddings to 128 dimensions

A computationally lightweight method that projects data into a lower-dimensional space using a randomly generated matrix. Based on the Johnson-Lindenstrauss lemma, which guarantees that pairwise distances are approximately preserved. Key properties:

• Extremely fast; requires no training or data analysis
• Data-oblivious; the projection matrix is independent of the input
• Often used with sparse random matrices for additional efficiency
• Preserves cosine similarity and Euclidean distance within a bounded distortion factor
• Ideal for rapid dimensionality reduction in streaming or resource-constrained environments

A linear technique that decomposes a multivariate signal into statistically independent, non-Gaussian components. Unlike PCA which decorrelates via variance, ICA maximizes statistical independence. Key properties:

• Assumes the observed data is a linear mixture of latent independent sources
• Useful for blind source separation and feature extraction
• Requires non-Gaussianity in the source signals to be identifiable
• Often applied after PCA whitening to reduce computational load
• Example: Separating mixed embedding signals into distinct semantic factors

A linear transformation that identifies the orthogonal axes of maximum variance in the data. PCA projects high-dimensional embeddings onto a lower-dimensional hyperplane while preserving as much global structure as possible.

• Computes the covariance matrix and extracts eigenvectors
• Optimal for linear data structures; fast and deterministic
• Commonly used to reduce embedding dimensions from 1024 to ~100 for visualization

A non-linear technique designed primarily for visualizing high-dimensional data in 2D or 3D. t-SNE converts Euclidean distances into conditional probabilities representing similarities, then minimizes the KL divergence between the high-dimensional and low-dimensional distributions.

• Excels at preserving local neighborhood structure
• Uses a heavy-tailed Student-t distribution to avoid the crowding problem
• Computationally intensive; not suitable for large-scale compression

A manifold learning technique that constructs a fuzzy topological representation of the data. UMAP balances the preservation of both local and global structure, making it superior to t-SNE for many tasks.

• Significantly faster than t-SNE, scaling to millions of points
• Preserves more global data structure than t-SNE
• Supports supervised and semi-supervised dimensionality reduction
• Often used as a general-purpose pre-processing step for clustering

A training paradigm that forces an embedding model to learn representations that are useful across multiple truncated dimensions. A Matryoshka embedding of size 1024 maintains semantic fidelity at 256, 128, or even 64 dimensions.

• Enables adaptive retrieval without retraining
• Allows developers to trade accuracy for storage cost dynamically
• Named after Russian nesting dolls, reflecting the nested structure of the learned vector

A lossy compression technique that decomposes the original high-dimensional vector space into a Cartesian product of lower-dimensional subspaces. Each subspace is independently quantized using a codebook of centroids.

• Dramatically reduces the memory footprint of vector indexes
• Enables efficient approximate distance computation via lookup tables
• A foundational component of the IVF-PQ index in FAISS

A computationally efficient linear method based on the Johnson-Lindenstrauss lemma, which states that high-dimensional points can be projected into a much lower-dimensional space while approximately preserving pairwise distances.

• Uses a randomly generated projection matrix
• Extremely fast and data-oblivious; no training required
• Often used as a lightweight alternative to PCA for very high-dimensional sparse data