Dimensionality Reduction

Principal Component Analysis (PCA) is a linear, unsupervised technique that identifies the orthogonal axes (principal components) of maximum variance in the data. It projects the data onto these new axes, ordered by the amount of variance they explain.

• Key Mechanism: Performs an eigendecomposition of the data covariance matrix.
• Primary Use: Data compression, noise reduction, and decorrelating features before other algorithms.
• Example: Reducing 100 sensor readings to 10 principal components that capture 95% of the signal variance.
• Limitation: Assumes linear relationships and is sensitive to feature scaling.

t-Distributed Stochastic Neighbor Embedding (t-SNE) is a non-linear, probabilistic technique designed primarily for visualization. It minimizes the divergence between two distributions: one measuring pairwise similarities in the high-dimensional space, and one in the low-dimensional space.

• Key Mechanism: Uses a Student-t distribution in the low-dimensional space to alleviate the "crowding problem."
• Primary Use: Visualizing high-dimensional clusters in 2D or 3D (e.g., word embeddings, cell populations).
• Critical Note: t-SNE is computationally intensive and non-deterministic; distances between clusters in the output are not meaningful.
• Best Practice: Use PCA for initial dimensionality reduction (e.g., to 50 components) before applying t-SNE for efficiency.

Uniform Manifold Approximation and Projection (UMAP) is a non-linear, manifold learning technique based on topological data analysis. It constructs a high-dimensional graph representation of the data and then finds a low-dimensional graph that is as similar as possible.

• Key Mechanism: Assumes data is uniformly distributed on a Riemannian manifold.
• Primary Use: A faster, more scalable alternative to t-SNE for visualization that often better preserves global structure. Also used for non-linear feature reduction for downstream tasks.
• Advantage: Generally faster than t-SNE and can be applied to larger datasets while preserving more of the global data topology.
• Parameter Sensitivity: Results can vary significantly with the n_neighbors parameter, which controls the balance between local and global structure preservation.

Linear Discriminant Analysis (LDA) is a supervised linear projection technique that maximizes the separation between multiple classes. It finds axes that maximize the between-class variance while minimizing the within-class variance.

• Key Mechanism: Solves a generalized eigenvalue problem derived from within-class and between-class scatter matrices.
• Primary Use: Feature extraction for classification tasks and as a preprocessing step to improve class separability.
• Key Difference from PCA: PCA is unsupervised and maximizes total variance; LDA is supervised and maximizes class separability.
• Constraint: The maximum number of components is number_of_classes - 1.

An Autoencoder is a neural network architecture trained in an unsupervised manner to reconstruct its input. The model is forced to learn a compressed, dense representation (the "code" or "latent space") in a bottleneck layer.

• Key Mechanism: Comprises an encoder network (compresses input to latent code) and a decoder network (reconstructs input from code). The loss is the reconstruction error (e.g., Mean Squared Error).
• Primary Use: Non-linear dimensionality reduction, anomaly detection (high reconstruction error), and generative modeling (with variational autoencoders).
• Advantage: Can learn complex, non-linear manifolds that linear methods like PCA cannot.
• Variants: Variational Autoencoders (VAEs) learn a probabilistic latent space, enabling data generation. Denoising Autoencoders are trained to reconstruct clean inputs from corrupted versions, learning robust features.

Random Projection is a computationally efficient, linear dimensionality reduction technique that relies on the Johnson-Lindenstrauss lemma. This lemma guarantees that pairwise distances between points are approximately preserved when projected onto a random lower-dimensional subspace.

• Key Mechanism: Projects the data using a random matrix whose entries are drawn from a distribution with zero mean and unit variance (e.g., Gaussian).
• Primary Use: Extremely fast initial dimensionality reduction for very high-dimensional data (e.g., text with 100k+ features) as a preprocessing step for other algorithms.
• Advantage: Speed and simplicity. It is often used as a first-pass reduction before applying a more expensive method like PCA.
• Practical Note: The quality of the projection depends on the target dimensionality, not the original dimensionality, making it highly scalable.

Dimensionality reduction acts as a lossy compressor for input features, drastically reducing the computational load during inference. Techniques like Principal Component Analysis (PCA) or autoencoders project raw sensor data (e.g., images, audio spectrograms) into a lower-dimensional latent space.

• Key Benefit: Enables complex models (like vision transformers) to run on microcontrollers by reducing the size of the initial feature vector that the model must process.
• Example: A 1024-pixel image can be projected to 64 principal components, reducing the first-layer matrix multiplication cost by ~94% before the main model even runs.

For RAG systems on edge devices, dimensionality reduction is essential for creating and searching vector embeddings of documents or context. High-dimensional embeddings from models like BERT are computationally expensive to store and compare.

• Process: Techniques like PCA or Uniform Manifold Approximation and Projection (UMAP) compress 768-dimensional embeddings into 128 or 64 dimensions.
• Impact: This allows a local vector database (e.g., on a smartphone) to perform fast semantic search with minimal memory footprint and latency, enabling factual, context-aware responses without cloud dependency.

In federated learning across edge devices, raw data never leaves the device. Dimensionality reduction is applied locally to create a standardized, compact representation before generating model updates.

• Privacy & Efficiency: Transmitting reduced-dimensional data summaries or gradients, rather than raw high-dimensional data (e.g., medical images), minimizes bandwidth use and reduces the risk of data reconstruction attacks.
• Use Case: Wearables aggregating health sensor data (heart rate, accelerometer, GPS) can extract core principal components and share only these for centralized model aggregation, preserving user privacy.

Understanding why a small model fails on an edge device is challenging. Dimensionality reduction techniques like t-SNE and UMAP are used to visualize high-dimensional model activations or failure cases in 2D/3D.

• Debugging Workflow: Engineers can cluster misclassified samples in the reduced space to identify data drift or under-represented edge cases in the training set.
• Practical Value: This visual diagnostic tool is crucial for iterative improvement of tinyML models deployed in the field, where logging raw high-dimensional data is infeasible.

For continual learning on edge devices, storing and processing high-dimensional raw data for training is impossible. Dimensionality reduction enables core-set selection and data distillation.

• Method: A large dataset is projected into a lower-dimensional space. A representative subset (core-set) that spans this space is selected for training.
• Result: The device can perform few-shot adaptation or fine-tuning using only this compact, informative subset, dramatically reducing memory and compute requirements for on-device training cycles.

Edge devices (e.g., robots, smart cameras) often have multiple sensors (LiDAR, camera, IMU). Dimensionality reduction aligns and fuses these heterogeneous, high-dimensional data streams into a unified, lower-dimensional representation.

• Technique: Canonical Correlation Analysis (CCA) or deep autoencoders find a shared latent space where data from different modalities are comparable.
• Application: This enables efficient multi-modal models on edge hardware, allowing an autonomous mobile robot to correlate visual features with spatial point clouds using less powerful hardware.

The process of using domain knowledge to create new, more informative input features from raw data. While dimensionality reduction simplifies existing features, feature engineering creates new ones to make patterns more learnable for models.

• Key Activities: Creating interaction terms, binning continuous variables, extracting date/time components, and applying domain-specific transformations.
• Goal: Improve model performance, interpretability, and efficiency by providing better signals than the raw data alone.
• Edge Relevance: Well-engineered features can reduce the need for complex models, making inference faster and more suitable for edge hardware.

A technique that reduces the numerical precision of data values (e.g., from 32-bit floating-point to 8-bit integers) to decrease memory footprint and computational cost.

• Mechanism: Maps a larger set of input values to a smaller set of output values, trading off some precision for efficiency.
• Application: Critical for processing high-dimensional data (like images or embeddings) on edge devices with limited memory bandwidth.
• Relation to Dimensionality Reduction: While quantization compresses the representation of each feature, dimensionality reduction reduces the number of features. They are often used together for maximum efficiency.

A data subset selection method that identifies a small, representative subset of a large dataset that preserves its essential geometric or statistical properties.

• Objective: Enable efficient model training on a fraction of the original data without significant performance loss.
• Methodology: Often uses algorithms like k-center or facility location to find points that best cover the data distribution.
• Strategic Use: In edge contexts, core-set selection can be performed on a server to create a minimal, high-quality training dataset before sending it to devices for on-device training or fine-tuning.

The process of removing redundant, noisy, or low-quality samples from a training dataset to improve training efficiency and model robustness.

• Rationale: Not all data is equally useful; some samples can be harmful or provide marginal learning benefit.
• Techniques: Include pruning by loss (removing easy-to-learn samples), by memorization, or by influence estimation.
• Complement to Dimensionality Reduction: Data pruning reduces the number of samples, while dimensionality reduction reduces the number of features. Both aim to create a more efficient, higher-signal dataset for training.

A machine learning paradigm where a model generates its own supervisory signal from the inherent structure of unlabeled data, typically by solving pretext tasks.

• Pretext Tasks: Include predicting masked parts of an input (masked language modeling), predicting the rotation of an image, or solving jigsaw puzzles.
• Output: The model learns a rich, general-purpose representation (an embedding) in a high-dimensional space.
• Downstream Dimensionality Reduction: These high-dimensional embeddings are prime candidates for techniques like PCA or UMAP to create 2D/3D visualizations for analysis or to produce lower-dimensional features for efficient downstream task models on edge devices.

A decentralized machine learning approach where a global model is trained across multiple edge devices holding local data samples, without exchanging the raw data itself.

• Privacy Benefit: Raw data never leaves the device; only model updates (gradients or weights) are shared.
• Communication Bottleneck: Transmitting high-dimensional model updates from many devices is expensive.
• Integration Point: Dimensionality reduction techniques can be applied to compress these updates or to reduce the dimensionality of the features learned on each device before aggregation, significantly reducing communication overhead in federated systems.