Model Inversion

The attack leverages the model's output confidence scores for each class to iteratively refine a synthetic input. By performing gradient descent on the input space rather than the model weights, an attacker minimizes the loss between the model's prediction for the synthetic input and a target class distribution. This process reveals the prototypical features the model associates with that class, such as reconstructing a face from a facial recognition model trained on private images.

In a white-box setting, the attacker has full access to model parameters and gradients, enabling precise reconstruction through direct optimization on the input. In a black-box setting, the attacker only queries the model API and observes confidence scores, using numerical approximation or zero-order optimization to estimate gradients. While white-box attacks produce higher-fidelity reconstructions, black-box attacks are more realistic for production APIs that expose per-class probabilities.

Model inversion is often paired with membership inference attacks to determine if a specific individual's data was in the training set. After reconstructing a class prototype, the attacker compares it against known samples using a shadow model trained on similar data distributions. If the reconstruction closely matches a specific individual rather than a generic class average, the model has effectively memorized that record, confirming membership and exposing the individual's private attributes.

The primary defense against model inversion is differential privacy (DP), which adds calibrated noise to the training process or output probabilities. By clipping gradients and injecting Gaussian noise during stochastic gradient descent, DP bounds the influence of any single training example on the final model. This prevents the model from memorizing individual records, forcing reconstructions to converge on generic, non-identifiable class averages rather than specific training samples.

While most research focuses on neural networks, model inversion also affects decision trees and regression models. For a decision tree trained on medical data, an attacker can reconstruct the path a specific patient's record took through the tree by observing the output, revealing sensitive attribute ranges. In linear regression, inverting the model matrix can directly solve for input features, exposing salary or health data when the model's coefficients are known.

A more constrained attack variant uses only the hard label prediction—the top-1 class without confidence scores—to perform inversion. The attacker trains a surrogate model on auxiliary data, then uses the target model's labels to guide an adversarial perturbation process. By observing how small input changes flip the predicted label, the attacker maps the decision boundary and reconstructs class features. This attack succeeds even when confidence scores are hidden, making it a critical threat for label-only APIs.

An attack that determines whether a specific data record was part of a model's training set. The adversary queries the target model and analyzes confidence scores, loss values, or output distributions to infer membership. This creates a significant privacy breach by exposing an individual's presence in sensitive datasets such as medical records or financial transactions. Shadow models are often trained to mimic the target's behavior and calibrate the attack. Membership inference is a precursor to more sophisticated extraction attacks.

A mathematical framework that provides a provable guarantee against model inversion and membership inference attacks. It works by injecting calibrated noise into training algorithms or query responses, ensuring that any single record's inclusion or exclusion has a statistically negligible effect on outputs. The privacy budget is controlled by the parameter epsilon (ε) — lower values provide stronger privacy at the cost of model utility. Implementations include DP-SGD for deep learning and the Gaussian mechanism for query answering.

A carefully perturbed input that appears normal to humans but causes a model to produce a high-confidence misclassification. While distinct from model inversion, adversarial examples share the common thread of exploiting model internals. In agent systems, adversarial inputs can trigger unauthorized actions or bypass safety filters. White-box attacks use gradient information, while black-box attacks rely on query access — the same access model required for inversion attacks. Defenses include adversarial training and input sanitization.

An attack on the training pipeline where an adversary injects malicious samples to corrupt the model's learned behavior. Backdoor poisoning embeds a trigger pattern that causes targeted misclassification at inference time. Availability poisoning degrades overall model performance. In the context of model inversion, a poisoned model may be more susceptible to leaking training data, as the corrupted decision boundaries can amplify memorization. Clean-label attacks are particularly insidious as they use correctly labeled, natural-looking samples.