Principal Component Analysis (PCA)

PCA performs an orthogonal transformation of the original feature space. This means the resulting principal components are linearly uncorrelated (orthogonal) and are ordered by the amount of variance they explain in the data.

• The transformation is defined by the eigenvectors of the data's covariance matrix.
• The first principal component (PC1) aligns with the direction of maximum variance.
• Each subsequent component captures the next highest variance under the constraint of being orthogonal to all preceding components.

The fundamental objective of PCA is variance maximization. It seeks a new coordinate system where the first axis captures the greatest spread in the data.

• The eigenvalues associated with each eigenvector quantify the variance explained by that component.
• The scree plot, which plots eigenvalues in descending order, helps visualize the contribution of each component and is used to select the optimal number of components to retain.
• Retaining components that capture, for example, 95% of the cumulative variance is a common heuristic for dimensionality reduction.

PCA is a linear dimensionality reduction technique. It assumes that the principal components are linear combinations of the original features.

• This makes it highly efficient and interpretable but limits its effectiveness on data with complex non-linear relationships.
• For non-linear manifolds, techniques like Kernel PCA or t-SNE are more appropriate.
• The linearity also means that the loadings (coefficients of the linear combination) can be directly inspected to understand which original features contribute most to each component.

PCA is computationally solved via Eigen-decomposition of the covariance matrix or, more commonly and robustly, via Singular Value Decomposition (SVD) of the mean-centered data matrix.

• SVD factorizes the data matrix X into U Σ V^T, where V contains the principal components (loadings) and Σ contains the singular values (related to eigenvalues).
• Using SVD is numerically more stable and avoids explicitly calculating the potentially large covariance matrix.
• This foundation makes PCA scalable and a standard component in numerical computing libraries.

PCA is sensitive to the scale of the input features. Features with larger ranges or variances will dominate the first principal components, even if they are less informative.

• Standardization (subtracting the mean and dividing by the standard deviation) is a critical preprocessing step when features are on different scales (e.g., height in cm vs. income in dollars).
• Without standardization, PCA will be biased toward high-variance features.
• When all features are naturally on the same scale (e.g., pixel intensities), mean-centering alone may be sufficient.

PCA's primary applications stem from its variance-preserving property:

• Data Compression & Noise Reduction: By projecting data onto the top k principal components, high-dimensional data can be represented in a lower-dimensional space with minimal loss of information. Components with small eigenvalues are often considered noise and can be discarded.
• Data Visualization: Projecting data onto the first two or three principal components allows for 2D/3D visualization of high-dimensional datasets, revealing clusters, outliers, and patterns.
• Feature Extraction & Decorrelation: The transformed features (principal component scores) are uncorrelated, which can benefit downstream machine learning models that assume feature independence.

Singular Value Decomposition (SVD) is the underlying matrix factorization algorithm that powers PCA. It decomposes any real or complex matrix into three constituent matrices: U (left singular vectors), Σ (diagonal matrix of singular values), and V^T (right singular vectors).

• In PCA, the principal components are the right singular vectors (V) of the centered data matrix.
• The singular values (Σ) correspond to the square roots of the eigenvalues of the covariance matrix, indicating the variance captured by each component.
• SVD is more numerically stable than directly computing eigenvectors and is the standard computational method for PCA in practice.

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a nonlinear dimensionality reduction technique primarily used for visualizing high-dimensional data in 2D or 3D. Unlike PCA, which preserves global variance, t-SNE focuses on preserving local neighborhoods.

• It converts high-dimensional Euclidean distances between data points into conditional probabilities representing similarities.
• A Student-t distribution is used in the low-dimensional map to alleviate the "crowding problem."
• Key difference from PCA: t-SNE is excellent for revealing cluster structure in visualizations but is computationally heavier, non-deterministic, and the low-dimensional axes are not interpretable as principal components.

Linear Discriminant Analysis (LDA) is a supervised dimensionality reduction method that finds axes that maximize the separation between multiple classes. While PCA is unsupervised and maximizes variance, LDA uses class labels to find projections that maximize between-class variance relative to within-class variance.

• Objective: Find a projection that best discriminates among predefined classes.
• Key difference from PCA: PCA finds directions of maximal variance in the data, which may not align with class separation. LDA is explicitly designed for classification tasks and can yield better feature spaces for classifiers when labels are available.
• It assumes data is normally distributed and classes share a common covariance matrix.

An Autoencoder is a type of neural network trained to reconstruct its input, creating a compressed representation in its bottleneck layer. This provides a nonlinear alternative to PCA for dimensionality reduction and feature learning.

• Architecture: Encoder network compresses input to a latent code (the bottleneck), decoder network reconstructs the input from this code.
• A linear autoencoder with a mean-squared error loss and no nonlinearities learns a subspace identical to that found by PCA.
• Advantage over PCA: Deep, nonlinear autoencoders can learn more complex, hierarchical manifolds in the data, making them powerful for tasks where data relationships are not linear.

Factor Analysis (FA) is a statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors. It is closely related to PCA but differs in its underlying model.

• Model: Assumes observed data is generated from latent factors plus independent, variable-specific noise (unique variance).
• Key difference from PCA: PCA treats all variance as potentially meaningful and seeks components that explain maximal total variance. FA explicitly models and separates common variance (shared by variables) from unique variance (noise). This makes FA more interpretable in psychometrics and social sciences for identifying latent constructs.

Kernel PCA is an extension of PCA that uses the kernel trick to perform nonlinear dimensionality reduction. It implicitly maps data into a higher-dimensional feature space where linear PCA is performed, allowing it to discover nonlinear manifolds.

• Process: A kernel function (e.g., Radial Basis Function) computes dot products in the high-dimensional space without explicitly performing the transformation.
• Use case: Effective for data where the principal manifold is not linear, such as concentric circles or spirals, where standard PCA would fail.
• Trade-off: Provides nonlinear capability but loses the interpretability of principal components as linear combinations of original features and can be computationally intensive for large datasets.