Model Inversion

In federated learning or collaborative training, an honest-but-curious server can reconstruct a client's private data batch directly from shared gradients. By optimizing a dummy input to produce gradients that match the true transmitted gradients, the attacker iteratively recovers pixel-perfect images or exact text sequences. Deep Leakage from Gradients (DLG) demonstrated this requires only a few hundred iterations of optimization to converge to the private input. Defense: Secure aggregation, differential privacy noise injection, or gradient compression.

When only API access to confidence scores is available, an attacker can formulate the inversion as a maximum likelihood estimation problem. By querying the model with candidate inputs and observing the output probabilities, the attacker iteratively refines a reconstruction. Fredrikson et al. demonstrated this by recovering recognizable faces from a facial recognition API using only the patient's name and the model's confidence vector. Defense: Return only hard labels, truncate confidence scores, or add calibrated noise.

Even without reconstructing raw data, an attacker can infer sensitive attributes absent from the input by exploiting correlations learned by the model. If a model predicts creditworthiness from non-sensitive features, an inversion attack can infer a protected attribute like race or gender by analyzing the model's decision boundary. This violates differential privacy and exposes the model to regulatory risk under GDPR and the EU AI Act. Defense: Adversarial training to remove spurious correlations, or formal fairness constraints.

Models that expose intermediate embeddings as a service are vulnerable to inversion attacks that decode the rich semantic content of the representation vector. An attacker trains a separate inversion decoder on an auxiliary dataset to map embeddings back to the input space. This is particularly dangerous for text embeddings used in RAG systems, where proprietary document chunks can be partially reconstructed. Defense: Dimensionality reduction, adding noise to embeddings, or restricting embedding API access.

Generative models like GANs and diffusion models can memorize and regurgitate training examples. A model inversion attack against a GAN's discriminator can recover training data by finding the latent code that maximizes the discriminator's confidence for a target class. Plug & Play Generative Networks demonstrated synthesizing images of a specific individual by inverting a pretrained generator. Defense: Deduplication of training data, differentially private training, or memorization auditing.

The most constrained attack surface—access to only the predicted hard label—still permits inversion. An attacker can probe the model's decision boundary by generating adversarial examples that cross it, revealing the geometry of the model's internal representation. By estimating the distance to the boundary from multiple directions, the attacker reconstructs a surrogate of the private input. Defense: Prediction throttling, query monitoring, or decision boundary smoothing.

A privacy attack that determines whether a specific data record was part of a model's training set by analyzing the model's confidence scores or internal activations. This exploits the tendency of models to be overconfident on memorized training examples.

• Shadow models are often trained to mimic the target model's behavior
• Metric: Area Under the Curve (AUC) of the attack classifier
• Defense: Differential Privacy and model regularization

A technique for training a simple linear classifier on top of a frozen model's internal representations to diagnose what information is encoded at a specific layer. It serves as a foundational tool for auditing whether sensitive attributes or private features are extractable.

• Probe location: Residual stream or MLP output
• Interpretation: High probe accuracy indicates linear separability of the concept
• Contrast with: Non-linear probing for complex encoded knowledge

A mathematical framework that provides provable guarantees against privacy attacks like model inversion by bounding the influence of any single training example on the model's output. It is the gold standard defense for training on sensitive data.

• Parameter ε (epsilon): The privacy budget; lower values mean stronger privacy
• Mechanism: Adding calibrated noise (e.g., Gaussian) to gradients
• Trade-off: Stronger privacy guarantees often reduce model utility

A method for identifying the specific hidden states within a transformer that are causally responsible for recalling a fact. It systematically restores clean activations from a corrupted forward pass to localize where knowledge is stored, revealing potential targets for extraction attacks.

• Process: Corrupt input embeddings, then iteratively restore states
• Finding: MLP layers in middle blocks are critical for factual recall
• Application: Understanding the mechanism behind knowledge extraction

A defensive technique for removing a specific linear concept direction, such as a protected attribute or private feature, from a model's representations. This prevents the concept from influencing downstream predictions or being recovered by an inversion attack.

• Method: Projecting activations onto the nullspace of the concept vector
• Tools: INLP (Iterative Nullspace Projection), R-LACE
• Goal: Ensure fairness and privacy without full retraining

The study of why models fail on imperceptibly perturbed inputs, which is closely related to model inversion. Understanding these vulnerabilities reveals how models represent decision boundaries and what information can be extracted by exploiting them.

• Phenomenon: Small input changes cause high-confidence misclassifications
• Link to Inversion: Adversarial gradients can be used to reconstruct inputs
• Defense: Adversarial training to harden the model's loss landscape