Principal Component Analysis

PCA performs a strictly linear transformation of the original feature space. It finds new axes (principal components) that are linear combinations of the original variables. This means it cannot capture complex, non-linear relationships within the data. For non-linear structures, techniques like Kernel PCA or UMAP are required.

• Mathematical Basis: Relies on eigenvalue decomposition of the covariance matrix or singular value decomposition (SVD) of the data matrix.
• Constraint: Inherently assumes the data's principal directions of variance are straight lines or planes.

The core optimization objective of PCA is to sequentially find orthogonal directions (principal components) that capture the maximum possible variance in the data. The first component points in the direction of greatest variance, the second captures the next greatest variance orthogonal to the first, and so on.

• Variance Explained: The ratio of a component's eigenvalue to the sum of all eigenvalues quantifies the proportion of total variance it explains.
• Dimensionality Reduction: Components with negligible explained variance (often corresponding to noise) are discarded, reducing dimensionality while preserving the most informative signal.

The principal components produced by PCA are orthogonal (perpendicular) to each other in the feature space. This orthogonality ensures the transformed features are uncorrelated (have zero covariance).

• Statistical Independence: While components are linearly uncorrelated, they are not necessarily statistically independent unless the data is multivariate Gaussian.
• Computational Benefit: Orthogonality simplifies downstream analysis and prevents multicollinearity issues in models that assume independent inputs.

PCA is highly sensitive to the scale and units of the input features. A feature with a large numerical range (e.g., revenue in dollars) will dominate the variance calculation and skew the first principal component toward that axis, even if other features are more informative.

• Mandatory Preprocessing: Data must be standardized (Z-score normalized) prior to PCA if features are on different scales. This gives each feature equal initial weight.
• Exception: If all features are naturally in the same units (e.g., pixels in an image), standardization may be optional.

PCA is an unsupervised technique; it requires no labeled data or target variable. It operates solely on the input feature matrix X. The algorithm is also deterministic: for a given dataset, the computed principal components (ignoring sign flips) will always be the same.

• No Guidance from Labels: The directions of maximum variance may not align with directions that best separate classes for a classification task. For labeled data, Linear Discriminant Analysis (LDA) is a supervised alternative.
• Reproducibility: The deterministic nature ensures pipeline reproducibility, a key concern in production ML systems.

PCA is a primary tool for combating the curse of dimensionality. In high-dimensional spaces, data becomes sparse, and distance metrics lose meaning. By projecting data onto a lower-dimensional subspace of principal components, PCA reduces noise, mitigates overfitting, and decreases computational cost.

• Eigenvalue Spectrum: A sharp drop in the scree plot (eigenvalues vs. component number) indicates a natural intrinsic dimensionality.
• Trade-off: The choice of how many components k to retain balances information retention (variance explained) against model complexity and storage. A common threshold is 95-99% of cumulative variance.

The overarching process of reducing the number of random variables (features) in a dataset. PCA is a specific linear technique within this category. The goal is to mitigate the curse of dimensionality, reduce noise, compress data, and enable visualization. Other methods include:

• t-Distributed Stochastic Neighbor Embedding (t-SNE): Non-linear, good for visualization.
• Uniform Manifold Approximation and Projection (UMAP): Preserves more global structure than t-SNE.
• Linear Discriminant Analysis (LDA): Supervised method that maximizes class separability.

A neural network architecture designed for unsupervised learning of efficient data codings. It learns to compress input data into a lower-dimensional latent representation (the bottleneck) and then reconstruct the output from it. Unlike PCA, autoencoders can learn non-linear transformations. They are used for:

• Dimensionality reduction and feature learning.
• Anomaly detection (poor reconstruction indicates an outlier).
• Denoising (training on corrupted inputs to output clean data).

A fundamental matrix factorization technique from linear algebra. PCA is typically computed using SVD on the centered data matrix. SVD factorizes any real or complex matrix into three matrices: (A = U \Sigma V^T), where (V) contains the principal components. Key differences:

• SVD operates directly on the data matrix.
• PCA is a specific application of SVD on covariance matrices for statistical variance maximization.
• SVD is also crucial for recommender systems (via collaborative filtering) and latent semantic analysis in NLP.

The mathematical core of PCA. The principal components are the eigenvectors of the data's covariance matrix. The corresponding eigenvalues represent the amount of variance captured by each component.

• Largest Eigenvalue: Corresponds to the first principal component (direction of maximum variance).
• Eigenvector Magnitude: Defines the direction of the new axis in the feature space.
• Variance Explained: The ratio of an eigenvalue to the sum of all eigenvalues shows the proportion of total variance a component captures.

The process of using domain knowledge to create or transform input variables. PCA is often applied after feature engineering to handle the resulting high-dimensional space. It serves as an automated method for creating new, uncorrelated features (the principal components). Synergies include:

• Using PCA on engineered polynomial features to avoid multicollinearity.
• PCA as feature extraction: The components themselves become new, more informative features for downstream models.
• Contrast with manual feature selection, which picks existing features rather than creating new ones.

A suite of techniques to reduce a model's computational and memory footprint. While PCA compresses data, model compression techniques target the algorithm. They are complementary strategies for efficient AI systems:

• Pruning: Removing insignificant weights from a neural network.
• Quantization: Reducing numerical precision of weights/activations (e.g., 32-bit to 8-bit).
• Knowledge Distillation: Training a small "student" model to mimic a large "teacher" model. Using PCA on input data can reduce feature dimensions, which directly shrinks the first layer of a model, contributing to overall compression.