Dimensionality Reduction

A linear transformation that projects data onto the directions of maximum variance, discarding low-variance components that often encode instance-specific noise. Eigenvectors corresponding to the smallest eigenvalues are truncated, creating an irreversible compression step. This acts as a natural low-pass filter on the data manifold, removing the high-frequency details that inversion attacks exploit.

• Reduces dimensionality by retaining top-k principal components
• Discarded components often contain identifiable features
• Linear operation makes privacy-utility trade-off analytically tractable

A generative model that learns a probabilistic latent space by encoding inputs into a distribution (mean and variance) and sampling from it during decoding. The KL divergence regularization term forces the latent distribution toward a standard Gaussian prior, creating a stochastic bottleneck that limits the mutual information between the latent code and the original input. This inherent stochasticity degrades reconstruction fidelity for unauthorized parties.

• Encodes inputs as distributions, not deterministic points
• Sampling operation introduces non-invertible randomness
• Beta-VAE variants increase disentanglement and privacy

A training objective formalized by Tishby et al. that seeks a latent representation Z that is maximally informative about the target Y while minimizing mutual information with the input X. By directly penalizing I(X;Z), the model is forced to discard irrelevant details, including identity-specific features. This creates a provable trade-off between task performance and inversion vulnerability.

• Minimizes I(X;Z) while maximizing I(Z;Y)
• Lagrangian formulation: min I(X;Z) - β I(Z;Y)
• Higher β values enforce stronger compression and privacy

A representation learning framework that maximizes the coding rate difference between the entire dataset and individual class-conditional distributions. This promotes representations that are both discriminative (classes are well-separated) and compressed (within-class variation is minimized). The compression term naturally suppresses instance-level details, making reconstructed inputs from inverted features blurry and unrecognizable.

• Optimizes rate distortion in feature space
• Within-class compression removes individual signatures
• Provides geometric guarantees on representation structure

A computationally lightweight technique that projects high-dimensional data into a lower-dimensional space using a random matrix whose entries are drawn from a Gaussian or sparse distribution. The Johnson-Lindenstrauss lemma guarantees that pairwise distances are approximately preserved, but the random projection is non-invertible without knowledge of the specific random matrix, which can be discarded after transformation.

• Uses a fixed random matrix for projection
• Preserves distance geometry for downstream tasks
• Discarding the projection matrix makes inversion computationally infeasible

Standard autoencoders learn a deterministic compressed representation by minimizing reconstruction error. For privacy, the bottleneck layer width is deliberately constrained to a dimension far smaller than the intrinsic dimensionality of the data. This forces the encoder to prioritize global structure over local details. The decoder cannot recover the discarded information, acting as a lossy compression defense.

• Bottleneck dimension controls privacy-utility trade-off
• Undercomplete architecture prevents identity mapping
• Can be combined with noise injection for stronger guarantees

A theoretical framework that formalizes the trade-off between compression and prediction. It seeks a latent representation T of input X that is maximally informative about target Y while minimizing mutual information with X. This naturally discards fine-grained details, creating a privacy funnel that starves inversion attacks of the high-frequency signals needed for reconstruction.

A representation learning principle that maximizes the coding rate difference between the whole dataset and individual classes. It promotes discriminative, low-dimensional subspace features that are inherently compressed. By learning a representation that is both diverse and compact, MCR2 provides a natural defense, as the compressed features lack the redundant information necessary for high-fidelity input reconstruction.

A defensive architecture that pre-processes inputs through a denoising or variational autoencoder (VAE) before they reach the primary classifier. The autoencoder learns a compressed latent bottleneck that strips away high-frequency, sensitive details. By projecting inputs into this lower-dimensional manifold, the defense obstructs inversion attacks, as the classifier never has access to the raw, reconstructible signal.

A stochastic regularization technique that randomly omits neural network units during training. This introduces uncertainty and sparsity in feature representations, effectively creating a dynamic, noisy bottleneck. The non-deterministic nature of dropout degrades the quality of reconstructed inputs from inversion attacks by preventing a stable, invertible mapping from forming in the latent space.

The primary threat that dimensionality reduction defenses aim to mitigate. These attacks invert intermediate feature representations or embeddings back into the original input space. By exploiting the semantic richness of deep features, an adversary can extract sensitive visual or textual information. Effective dimensionality reduction minimizes the mutual information between these features and the raw input, directly countering the attack's objective.

A distributed training paradigm where a model is partitioned between a client and server. The client only shares intermediate activations (smashed data) from a bottleneck layer, not raw data or full gradients. This architectural choice is a form of dimensionality reduction, as the shared representation is a compressed, non-invertible projection that drastically reduces the attack surface for gradient inversion and feature reconstruction.