Dimensionality Reduction

Principal Component Analysis (PCA) is a linear, unsupervised technique that identifies the orthogonal axes (principal components) of maximum variance in the data. It projects the original high-dimensional points onto a lower-dimensional subspace defined by the top-k eigenvectors of the data covariance matrix.

• Linear Transformation: Applies a rigid rotation and scaling to the data.
• Variance Preservation: The first principal component captures the most variance, the second captures the next most, and so on.
• Common Use: Preprocessing for other algorithms, noise filtering, and initial data exploration. It is computationally efficient via Singular Value Decomposition (SVD).

t-SNE is a nonlinear, probabilistic technique designed primarily for visualization. It converts high-dimensional Euclidean distances between data points into conditional probabilities representing similarities. It then constructs a low-dimensional map where these probabilities are preserved using a Student's t-distribution to mitigate crowding.

• Nonlinear Mapping: Excels at revealing local cluster structure and manifold geometry.
• Visualization Focus: Best for 2D or 3D plots to explore data clusters; the absolute positions and distances between clusters in the output are not interpretable.
• Computational Cost: Iterative and relatively slow for large datasets; results can vary with different initializations.

UMAP is a nonlinear, graph-based technique grounded in Riemannian geometry and algebraic topology. It assumes data is uniformly distributed on a Riemannian manifold, constructs a fuzzy topological representation of the high-dimensional data, and then optimizes a low-dimensional layout to be as topologically similar as possible.

• Preserves Structure: Aims to maintain both local and global structure better than t-SNE.
• Speed & Scalability: Often faster than t-SNE and can be applied to larger datasets.
• Flexible Applications: Used for visualization, but its embeddings can also be useful as inputs for downstream clustering or classification tasks.

An Autoencoder is a neural network-based, nonlinear method trained to reconstruct its input. It learns a compressed representation (the embedding in the 'bottleneck' layer) by forcing the network through a lower-dimensional latent space.

• Neural Architecture: Consists of an encoder (compresses input to latent code) and a decoder (reconstructs input from code).
• Learned Compression: The model learns data-specific, often semantic, features for reduction.
• Variants: Variational Autoencoders (VAEs) learn a probabilistic latent space, enabling generative sampling. Denoising Autoencoders learn robust representations from corrupted inputs.

Linear Discriminant Analysis is a supervised linear technique that projects data onto axes that maximize the separation between predefined classes. It finds directions that maximize the ratio of between-class variance to within-class variance.

• Supervised Method: Requires class labels for training.
• Class Separation Goal: Aims for optimal class discriminability in the reduced space, unlike PCA which focuses on variance alone.
• Common Use: Often used as a preprocessing step for classification tasks, reducing dimensions while enhancing class boundaries.

Random Projection is a computationally simple, linear technique based on the Johnson-Lindenstrauss lemma. It projects data onto a random lower-dimensional subspace using a matrix with random entries (e.g., Gaussian or sparse binary).

• Theoretical Guarantee: The Johnson-Lindenstrauss lemma ensures pairwise distances are approximately preserved with high probability.
• Extreme Speed: The projection matrix is not data-dependent, making it extremely fast to compute.
• Use Case: Ideal for very high-dimensional data as an initial, drastic reduction or for privacy-preserving transformations where the original data structure should be obscured.

Principal Component Analysis (PCA) is a linear dimensionality reduction technique that identifies the orthogonal axes (principal components) of maximum variance in the data. It projects the high-dimensional data onto a lower-dimensional subspace defined by these components.

• Key Mechanism: Uses eigendecomposition of the data covariance matrix.
• Primary Use: Data compression, noise reduction, and exploratory data analysis.
• Limitation: Assumes linear relationships within the data, which may not capture complex, nonlinear structures present in modern embeddings.

t-SNE is a nonlinear, probabilistic technique designed primarily for visualizing high-dimensional data in two or three dimensions. It converts similarities between data points into joint probabilities and tries to minimize the Kullback–Leibler divergence between the high-dimensional and low-dimensional probability distributions.

• Key Feature: Excels at preserving local structure and revealing clusters.
• Common Pitfall: The visualizations are stochastic; different runs can produce different layouts, and global structure is often not preserved.
• Typical Application: Visualizing word, sentence, or image embeddings to understand cluster formation.

UMAP is a general-purpose nonlinear dimensionality reduction algorithm based on manifold learning and topological data analysis. It assumes data is uniformly distributed on a Riemannian manifold and constructs a fuzzy topological representation to find a low-dimensional equivalent.

• Advantages over t-SNE: Often faster, better at preserving global data structure, and can be used for more than just visualization (e.g., as a pre-processing step).
• Core Parameter: n_neighbors, which balances the preservation of local versus global structure.
• Widespread Use: The current de facto standard for creating publication-quality visualizations of embedding spaces.

Singular Value Decomposition is a fundamental matrix factorization technique that decomposes a matrix into three constituent matrices. In the context of dimensionality reduction, it is the computational foundation for methods like PCA and Latent Semantic Analysis (LSA).

• Mathematical Definition: For a matrix A, SVD finds A = U Σ V^T, where U and V are orthogonal matrices and Σ is a diagonal matrix of singular values.
• Dimensionality Reduction: By keeping only the top k singular values and corresponding vectors (truncated SVD), one obtains a rank-k approximation of the original matrix, effectively reducing its dimensionality.
• Application: Directly used in collaborative filtering, topic modeling, and compressing term-document matrices.

An autoencoder is a neural network architecture used for unsupervised learning of efficient codings. It is trained to reconstruct its input, forcing a compressed bottleneck layer (the latent space) to learn a meaningful, lower-dimensional representation.

• Architecture: Consists of an encoder network that maps input to the latent code and a decoder network that reconstructs the input from the code.
• Variants: Variational Autoencoders (VAEs) learn a probabilistic latent space, enabling generative capabilities. Denoising Autoencoders are trained to reconstruct clean input from corrupted versions, learning robust features.
• Use Case: Nonlinear dimensionality reduction where deep feature learning is required, often outperforming linear methods on complex data like images or sequential text.

Manifold learning is a class of unsupervised machine learning techniques based on the assumption that high-dimensional data lies on or near a lower-dimensional, nonlinear manifold embedded within the high-dimensional space. The goal is to learn this intrinsic geometry.

• Core Principle: While PCA handles linear subspaces, manifold learning algorithms (like Isomap, LLE, and UMAP) model curved surfaces.
• Isomap (Isometric Mapping): Preserves geodesic distances (distances along the manifold) rather than straight-line Euclidean distances.
• LLE (Locally Linear Embedding): Models each data point as a linear combination of its nearest neighbors and seeks a low-dimensional representation that preserves these local linear relationships.
• Significance: Provides the theoretical foundation for most modern, effective nonlinear dimensionality reduction techniques.