Principal Component Analysis (PCA)

PCA performs an orthogonal linear transformation of the data. The resulting principal components are new, uncorrelated axes (vectors) that are mutually perpendicular (orthogonal) in the feature space. This orthogonality ensures that each component captures a distinct, non-redundant direction of variance in the data. The transformation is represented by a rotation (and possibly reflection) of the original coordinate system to align with the directions of maximum variance.

The core objective of PCA is to find a sequence of orthogonal directions that maximize the captured variance. The first principal component is the direction along which the projection of the data has the largest possible variance. Each succeeding component is then chosen to be orthogonal to the previous ones and to have the maximum possible remaining variance. This property makes PCA optimal for data reconstruction in the mean-squared error sense under linear projection constraints.

PCA is fundamentally solved via eigenvalue decomposition (or singular value decomposition). The steps are:

• Center the data (subtract the mean).
• Compute the covariance matrix of the centered data.
• Perform eigenvalue decomposition of the covariance matrix: Cov = VΛVᵀ.
• The columns of V are the eigenvectors (principal components).
• The diagonal entries of Λ are the corresponding eigenvalues, which represent the variance captured by each component. The eigenvectors define the new axes, and their eigenvalues indicate their importance.

Dimensionality reduction is achieved by truncating (discarding) components associated with the smallest eigenvalues. One retains only the top-k principal components that capture a sufficient proportion of the total variance. The explained variance ratio for a component is its eigenvalue divided by the sum of all eigenvalues. A common practice is to choose k such that the cumulative explained variance exceeds a threshold (e.g., 95% or 99%). This projects the original n-dimensional data onto a k-dimensional subspace (k < n).

A direct consequence of the orthogonal transformation is that the new features (the principal component scores) are linearly uncorrelated. While the original features may have been highly correlated, their projections onto the principal axes have zero covariance. This decorrelation simplifies subsequent analysis (e.g., regression) by eliminating multicollinearity. However, it's crucial to note that PCA only removes linear correlations; non-linear dependencies may persist in the reduced data.

PCA is not scale-invariant. The covariance matrix, and therefore the resulting principal components, are heavily influenced by the units and scales of the original features. A feature with a large numerical range (e.g., revenue in dollars) will dominate the first component compared to a feature with a small range (e.g., a percentage). Therefore, standardization (subtracting the mean and dividing by the standard deviation to create z-scores) is a critical preprocessing step when features are on different scales, ensuring each contributes proportionally to the analysis.

PCA is a primary tool for visualizing high-dimensional datasets by projecting them onto their first two or three principal components. This allows data scientists to identify clusters, outliers, and underlying patterns in a 2D or 3D scatter plot.

• Key Process: Data is centered, and the covariance matrix is decomposed. The top 2-3 eigenvectors define the projection plane.
• Example: Visualizing a dataset with 50 features (e.g., gene expression levels) becomes impossible directly. PCA reduces it to 2 dimensions, revealing patient subgroups.
• Limitation: The visualization is an approximation; variance lost in discarded components may contain meaningful signal.

PCA acts as a linear filter that separates signal from noise by assuming the signal has higher variance and lies in a lower-dimensional subspace. Components with small eigenvalues are often associated with noise and can be discarded.

• Mechanism: After decomposition, the original data is reconstructed using only the top-k principal components: (\tilde{X} = X W_k W_k^T), where (W_k) contains the top-k eigenvectors.
• Real-World Use: In electroencephalography (EEG) or financial time series, the first few PCs capture the dominant market trends or neural oscillations, while later components represent high-frequency noise.
• Trade-off: Aggressive truncation can remove subtle but important signals.

Before training a machine learning model, PCA is used to reduce the number of input features (dimensionality reduction). This decreases training time, reduces memory usage, and can mitigate the curse of dimensionality and overfitting, especially for models like logistic regression or SVMs.

• Procedure: Fit PCA on the training set, transform both training and test sets using the learned components.
• Critical Consideration: The transformed features (principal components) are linear combinations of all original features and are uncorrelated. This orthogonalization can improve numerical stability for some algorithms.
• Caution: The interpretability of original features is lost. It is generally applied to continuous, normalized data.

In computer vision, PCA is the basis for Eigenfaces, a seminal face recognition technique. An image (e.g., 64x64 pixels) is treated as a 4096-dimensional vector. PCA identifies the principal components of a dataset of face images—these are the "eigenfaces." Any face can then be approximated as a weighted sum of these eigenfaces.

• Compression: Storing only the weights for the top-k eigenfaces compresses the image data significantly.
• Application: Beyond faces, this technique applies to any set of aligned images (e.g., medical imaging, handwritten digits).
• Foundation: This application directly demonstrates PCA's role in finding the optimal basis for representing a specific data distribution.

In statistical regression, multicollinearity (high correlation among predictor variables) inflates standard errors and makes coefficient estimates unstable. PCA eliminates this issue by transforming the original correlated predictors into a set of uncorrelated principal components, which are then used as the new regressors.

• Process: Perform PCA on the predictor matrix. Use the principal component scores as independent variables in the regression (Principal Component Regression, PCR).
• Advantage: Guarantees orthogonal, non-collinear inputs.
• Drawback: Interpretation relates to components, not original business metrics, requiring a post-hoc analysis to map component importance back to original features.

PCA can be used to model "normal" data. By projecting data onto the principal subspace and then reconstructing it, the reconstruction error serves as an anomaly score. Data points that are poorly reconstructed (high error) deviate from the correlation structure learned by PCA and are flagged as anomalies.

• Method: A model is fit on normal data. For a new sample, compute (\text{error} = ||x - \tilde{x}||^2), where (\tilde{x}) is the PCA reconstruction. A high error indicates a potential anomaly.
• Use Case: Detecting fraudulent transactions, defective products in manufacturing, or network intrusions where anomalies violate the dominant correlation patterns.
• Characteristic: This is an unsupervised detection method, requiring no labeled anomaly data for training.

Singular Value Decomposition (SVD) is the fundamental matrix factorization that underpins PCA. For any real matrix (X), SVD factors it as (X = U \Sigma V^T), where (U) and (V) are orthogonal matrices containing the left and right singular vectors, and (\Sigma) is a diagonal matrix of singular values.

• In PCA, the principal components are the right singular vectors (V) (or the eigenvectors of the covariance matrix (X^TX)).
• The singular values in (\Sigma) correspond to the square roots of the eigenvalues, indicating the variance captured by each component.
• SVD is numerically stable and is the preferred computational method for performing PCA on centered data.

Eigenvalue Decomposition is the factorization of a square, diagonalizable matrix into its eigenvectors and eigenvalues. For PCA, the core computation is the eigendecomposition of the sample covariance matrix (C = \frac{1}{n-1} X^T X).

• The decomposition yields (C = V \Lambda V^T), where the columns of (V) are the eigenvectors (principal components) and (\Lambda) is a diagonal matrix of eigenvalues.
• The eigenvalues represent the variance of the data projected onto the corresponding eigenvector.
• This is mathematically equivalent to SVD on the centered data matrix, as (C = V \frac{\Sigma^2}{n-1} V^T).

Low-Rank Approximation is the process of representing a matrix using a smaller, approximate version with a reduced rank (k). PCA is a specific, optimal form of low-rank approximation for centered data.

• The Eckart–Young theorem proves that the best rank-(k) approximation of a matrix (in Frobenius norm) is given by its Truncated SVD, which retains only the top (k) singular values and vectors.
• In PCA, projecting data onto the top (k) principal components and reconstructing it yields the optimal rank-(k) approximation that minimizes reconstruction error.
• This is the mechanism behind PCA's dimensionality reduction and data compression capabilities.

Variance and Covariance are the statistical quantities PCA explicitly optimizes. PCA finds directions that maximize variance (spread) of the projected data, under the constraint that directions are orthogonal and have zero covariance with each other.

• The covariance matrix is symmetric and positive semi-definite. Its diagonal entries are feature variances, and off-diagonals are covariances between features.
• By diagonalizing this matrix (via eigendecomposition), PCA transforms the data into a new coordinate system where the new features (PCs) have maximal variance and zero covariance.
• The total variance in the data is preserved and equal to the sum of all eigenvalues; the proportion explained by a component is its eigenvalue divided by this sum.

Dimensionality Reduction is the broader machine learning task of reducing the number of random variables under consideration. PCA is the most common linear, unsupervised technique for this purpose.

• Goal: Transform high-dimensional data to a meaningful lower-dimensional representation, often for visualization, compression, or as a preprocessing step for other algorithms.
• Linear vs. Nonlinear: PCA performs a linear projection. Nonlinear techniques like t-SNE or UMAP capture more complex manifolds but are not based on variance maximization.
• Feature Extraction vs. Selection: PCA creates new, transformed features (principal components), unlike feature selection methods which choose a subset of the original features.

Whitening, or Zero-phase Component Analysis (ZCA), is a closely related transformation that decorrelates and standardizes the data. While PCA rotates data to align with axes of maximum variance, ZCA whitening rotates it back after scaling to preserve the original feature space orientation.

• Process: After PCA rotation, each component is scaled so its variance equals 1 (divided by the square root of its eigenvalue). This results in a whitened dataset with identity covariance matrix.
• ZCA: A specific whitening transform that applies an additional rotation back to the original basis, minimizing the Euclidean distance to the original data. It is often used in image processing as a preprocessing step.
• Difference: PCA yields orthogonal, variance-ordered components. Whitening yields uncorrelated components with unit variance, but not necessarily ordered.