Model Inversion

The attacker exploits the relationship between a model's learned parameters and its training data. By starting with random noise and iteratively optimizing an input to maximize the model's confidence for a target class, the attacker reconstructs a prototypical representation.

• White-box access to model weights and gradients is typically required
• Uses gradient descent on the input space rather than the parameter space
• Reconstructs a class representative that the model considers most likely
• Effective against softmax classifiers and facial recognition models
• The reconstructed image often resembles a blurred but recognizable version of the training class average

Even with black-box API access, an attacker can perform model inversion by querying the model with candidate inputs and observing the confidence scores returned. The attacker iteratively refines inputs to maximize the target class probability.

• Requires only prediction API access with confidence scores
• Uses evolutionary algorithms or hill-climbing to optimize inputs
• Exploits the overconfidence of poorly calibrated models
• Particularly effective when the model outputs full softmax distributions
• Defenses include limiting API output to hard labels only

In federated learning settings, model inversion can extract private features from shared gradient updates. An honest-but-curious server can reconstruct training data by analyzing the gradients transmitted during collaborative training.

• Exploits gradient leakage from shared model updates
• Reconstructs pixel-level data from gradient information
• Deep Leakage from Gradients (DLG) is a seminal attack in this category
• Defenses include secure aggregation and gradient perturbation
• The attack is amplified when using small batch sizes during training

Modern attacks leverage a pretrained GAN or diffusion model as a strong prior. The attacker searches the generative model's latent space to find an input that the target classifier assigns high confidence, producing photorealistic reconstructions.

• Uses a StyleGAN or similar generator as a natural image prior
• Optimizes in latent space rather than pixel space for realism
• Produces high-fidelity, recognizable faces from facial recognition models
• Knowledge distillation from the target model can amplify the attack
• Defenses include differential privacy during training to limit memorization

A related form of model inversion infers sensitive attributes about individuals in the training data without reconstructing full inputs. The attacker uses the model's outputs to determine demographic or private characteristics.

• Targets sensitive attributes like race, gender, or health status
• Exploits correlations learned by the model between features
• Does not require reconstructing the full training sample
• Fredrikson et al. demonstrated this against pharmacogenetic models
• Mitigation requires fairness constraints and adversarial debiasing

Defending against model inversion requires a layered approach combining differential privacy, regularization, and output restriction. No single defense is sufficient against all inversion variants.

• DP-SGD adds calibrated noise during training to limit memorization
• Temperature scaling smooths output confidences to reduce exploitability
• Prediction throttling limits the number of API queries per user
• Information bottleneck training naturally compresses sensitive features
• PATE aggregates teacher ensembles with noisy voting for privacy guarantees