Dimensionality Reduction

A linear transformation that identifies orthogonal axes of maximum variance. PCA projects data onto the top-k eigenvectors of the covariance matrix, minimizing reconstruction error.

• Mechanism: Singular Value Decomposition (SVD) on centered data.
• Use Case: Preprocessing for visualization or noise filtering before ANN indexing.
• Limitation: Assumes linear relationships; fails to capture non-linear manifolds.

Projects vectors into a lower-dimensional space using a randomly generated matrix. Based on the Johnson-Lindenstrauss lemma, it preserves pairwise distances with high probability.

• Advantage: Extremely fast and data-oblivious; no training required.
• Use Case: A lightweight alternative to PCA for massive, streaming datasets.
• Trade-off: Slightly higher distortion than optimized linear methods.

A non-linear technique focused on preserving local neighborhoods for visualization. t-SNE converts high-dimensional Euclidean distances into conditional probabilities representing similarities.

• Mechanism: Minimizes KL divergence between high-dim and low-dim joint probabilities.
• Use Case: Exploratory data analysis of embedding clusters in 2D/3D plots.
• Warning: Non-convex and computationally heavy; not suitable as a general indexing preprocessor.

A manifold learning technique that balances global structure and local neighborhood preservation. UMAP constructs a fuzzy topological representation of the data.

• Advantage: Faster than t-SNE and better preserves global structure.
• Mechanism: Assumes data lies on a Riemannian manifold; optimizes cross-entropy of fuzzy simplicial sets.
• Use Case: Pre-indexing dimensionality reduction where non-linear relationships dominate.

Neural networks trained to copy input to output through a bottleneck layer. The latent space representation serves as a learned, non-linear dimensionality reduction.

• Architecture: Encoder compresses; decoder reconstructs.
• Variants: Variational Autoencoders (VAEs) for generative modeling; Denoising Autoencoders for robustness.
• Use Case: Domain-specific compression where linear methods underperform.

Selecting a subset of the original dimensions rather than creating new synthetic features. Variance Thresholding removes low-variance features; Mutual Information selects features with the strongest statistical dependence on the target.

• Advantage: Maintains interpretability of original features.
• Use Case: Sparse retrieval pipelines where feature semantics must be preserved.
• Contrast: Unlike PCA, no information is blended across dimensions.

The phenomenon where data becomes increasingly sparse as dimensionality grows, causing distance metrics to lose discriminative power. In high-dimensional spaces, the ratio between the nearest and farthest neighbor converges to 1, making all points appear equidistant. This directly motivates dimensionality reduction: without it, exact nearest neighbor search becomes computationally infeasible and semantically meaningless.

A linear transformation technique that projects data onto orthogonal axes of maximum variance. PCA computes the eigenvectors of the covariance matrix and retains only the top-k components, preserving the global structure while discarding noise. It is widely used as a baseline for reducing embedding dimensions before indexing, though it assumes linear relationships and may underperform on nonlinear manifolds.

A computationally lightweight dimensionality reduction method based on the Johnson-Lindenstrauss lemma, which proves that pairwise distances can be preserved with high probability when projecting into a lower-dimensional space using a random matrix. Unlike PCA, random projection requires no training and is data-oblivious, making it ideal for streaming or dynamic datasets where recalculating principal components is impractical.

A vector compression technique that decomposes high-dimensional vectors into smaller subvectors, quantizing each independently using a distinct codebook. While primarily a compression method, PQ effectively performs dimensionality reduction by representing vectors as short codes. The tradeoff is quantization error: the distortion introduced when mapping continuous vectors to discrete centroids.

t-Distributed Stochastic Neighbor Embedding is a nonlinear technique designed specifically for visualization. It converts pairwise Euclidean distances into conditional probabilities representing similarities, then minimizes the KL divergence between high-dimensional and low-dimensional distributions. t-SNE excels at revealing local clusters and manifolds but is computationally expensive and non-deterministic, making it unsuitable for indexing pipelines.

Uniform Manifold Approximation and Projection constructs a fuzzy topological representation of the data, then optimizes a low-dimensional embedding to preserve both local and global structure. UMAP is significantly faster than t-SNE and better preserves global relationships, making it increasingly popular for exploratory analysis of embedding spaces before selecting an appropriate dimensionality for ANN indexing.