Dimensionality Reduction

A linear technique that orthogonally transforms data into a new coordinate system where the greatest variance lies on the first coordinate (first principal component). PCA computes the eigenvectors of the covariance matrix to identify the directions of maximum spread.

• Mechanism: Eigendecomposition of the covariance matrix or Singular Value Decomposition (SVD).
• Use Case: Whitening embeddings to remove anisotropy, initializing other algorithms, and visualizing global structure.
• Limitation: Fails to capture non-linear relationships in complex semantic manifolds.

A non-linear technique focused on preserving local neighborhoods. t-SNE converts high-dimensional Euclidean distances into conditional probabilities representing similarities, then minimizes the Kullback-Leibler divergence between these and a Student-t distribution in the low-dimensional space.

• Perplexity: A critical hyperparameter that loosely determines the number of effective nearest neighbors.
• Use Case: Visualizing distinct clusters in word embeddings or image features.
• Limitation: High computational cost (O(n²)), non-convex optimization leading to different results per run, and poor preservation of global geometry.

A manifold learning technique that constructs a fuzzy topological representation of the high-dimensional data using simplicial sets, then optimizes a low-dimensional representation to be as structurally similar as possible using cross-entropy.

• Advantage: Preserves both local and global structure significantly better than t-SNE.
• Performance: Scales much faster than t-SNE, making it suitable for large embedding datasets.
• Use Case: Preprocessing for clustering, general-purpose visualization of vector spaces, and feature engineering for downstream models.

A neural network architecture trained to copy its input to its output through a bottleneck layer. The encoder compresses the input into a latent code, and the decoder reconstructs the original data from this code.

• Variational Autoencoders (VAEs): Enforce a continuous, structured latent space by modeling it as a probability distribution, enabling generative sampling.
• Use Case: Non-linear feature extraction, denoising embeddings, and learning compressed representations for semantic hashing.
• Advantage: Can learn highly complex, non-linear mappings specific to the data distribution.

A computationally simple technique based on the Johnson-Lindenstrauss lemma, which states that high-dimensional points can be projected into a much lower-dimensional space while approximately preserving pairwise distances.

• Mechanism: Multiply the original data matrix by a random matrix whose rows are sampled from a Gaussian or sparse distribution.
• Use Case: Extremely fast dimensionality reduction for high-volume streaming data or as a preprocessing step before applying more expensive algorithms like PCA.
• Advantage: Data-oblivious and computationally very cheap, with strong theoretical guarantees on distance distortion.

A linear technique that separates a multivariate signal into additive, statistically independent non-Gaussian components. Unlike PCA which decorrelates signals, ICA minimizes mutual information to find the most independent source signals.

• Mechanism: Maximizes non-Gaussianity (kurtosis or negentropy) of the projected components.
• Use Case: Blind source separation, such as isolating distinct semantic features or disentangling latent factors in word embeddings.
• Limitation: Assumes the underlying sources are non-Gaussian and statistically independent.