Data Reconstruction Attack

In federated learning and distributed training, raw gradients shared between nodes encode substantial information about local training batches. An honest-but-curious server can reconstruct pixel-accurate images or token-for-token text by optimizing dummy inputs to match observed gradients.

• Deep Leakage from Gradients (DLG): Iteratively aligns dummy gradients with true gradients using L-BFGS optimization
• Inverting Gradients (IG): Recovers high-fidelity images by minimizing cosine similarity between generated and actual gradient vectors
• Impact: Achieves >95% pixel-wise reconstruction fidelity on CIFAR-10 and MNIST datasets under standard training configurations

Attackers exploit a trained generative model's own decoder or a separately trained GAN to map from low-dimensional latent representations back to the high-dimensional training distribution. By optimizing latent vectors to maximize confidence scores for a target class, sensitive class-representative samples emerge.

• Class-representative inversion: Recovers prototypical faces for identity-labeled classifiers
• StyleGAN-based inversion: Leverages pre-trained generators as strong natural image priors to produce photorealistic reconstructions
• Real-world example: Reconstructed recognizable faces from a facial recognition model trained on the AT&T Faces dataset with only black-box API access

Black-box attacks that require only model prediction vectors. By observing how confidence scores change across carefully crafted queries, attackers iteratively hill-climb toward the training data manifold. Higher confidence indicates proximity to memorized training points.

• Model inversion attack (Fredrikson et al.): Recovers genomic markers and facial images using only predicted confidence scores
• Decision boundary probing: Maps the model's internal decision surface to infer training point locations
• Mitigation challenge: Even top-1 label-only access can leak information through decision boundary geometry

Large overparameterized models inadvertently memorize rare or unique training sequences. Attackers trigger this memorization through specific prompting or by identifying outlier high-confidence predictions that correspond to verbatim training data.

• Extractable memorization: Language models can be prompted to regurgitate training data containing names, emails, and code snippets verbatim
• k-Eidetic memorization: Diffusion models memorize and reproduce near-identical copies of individual training images under specific conditioning
• Canary extraction: Inserted canary sequences in training data are recoverable with as few as 1-10 occurrences, demonstrating the privacy risk of rare data points

Even when full reconstruction fails, attackers can recover sensitive attributes of training subjects by inverting intermediate layer embeddings. This partial reconstruction reveals demographic, medical, or behavioral characteristics without needing the raw input.

• Embedding inversion attacks: Recover sensitive attributes like race, gender, or disease status from penultimate layer representations
• Feature vector leakage: Collaborative filtering embeddings encode individual user preferences recoverable through matrix factorization attacks
• Cross-modal leakage: Text embeddings from clinical notes can reveal patient identity when aligned with public demographic datasets

For sequential models such as RNNs and Transformers trained on time-series or language data, attackers reconstruct entire sequences by autoregressively sampling from the model while conditioning on partial or adversarial prefixes.

• Next-token probability analysis: Language model output distributions reveal training data patterns through statistical frequency analysis
• Trajectory recovery: Mobility models trained on GPS traces can reconstruct individual movement patterns from partial query sequences
• Differential sequence attack: Comparing output distributions between models trained with and without a target sequence isolates its influence for reconstruction

A class of attacks that reconstructs representative features of a target class by iteratively querying a model's confidence scores. Unlike gradient-based reconstruction, model inversion exploits the model's output API to generate a prototypical image—such as a recognizable face—that maximizes the activation of a specific class node. This technique is particularly effective against high-dimensional classifiers in medical imaging and facial recognition systems.

An attack that recovers raw training data from shared parameter gradients in distributed learning. By analyzing the gradients transmitted during federated learning updates, an honest-but-curious server can reconstruct pixel-level images or token-level text. Key mechanisms include:

• Analytical gradient inversion for fully connected layers
• Optimization-based reconstruction using cosine similarity loss
• Recursive gradient matching for transformer architectures

A privacy breach that infers sensitive attributes not explicitly present in the training data by exploiting correlations learned by the model. For example, a model trained on anonymized medical records may inadvertently reveal genetic predispositions or socioeconomic status through its predictions. This attack leverages collateral information and statistical dependencies embedded in the model's decision boundaries.

The primary mathematical defense against data reconstruction attacks. Differential privacy provides a formal guarantee that the output of an algorithm is statistically indistinguishable whether or not any single individual's data is included. Implementation involves:

• Calibrated noise injection via Gaussian or Laplacian mechanisms
• Privacy budget (ε) tracking to bound total information leakage
• Clipping gradients to limit individual sample influence during training

A phenomenon where generative models inadvertently store and reproduce exact training examples rather than learning generalizable distributions. Large language models and diffusion models are particularly susceptible—researchers have extracted verbatim PII, code snippets, and copyrighted text through strategic prompting. This represents a passive form of data reconstruction that requires no gradient access.

A cryptographic protocol enabling multiple parties to jointly compute a function over their private inputs without revealing those inputs to each other. In the context of ML, SMPC allows private inference where neither the model owner nor the data owner exposes their assets. Techniques include:

• Garbled circuits for boolean operations
• Secret sharing for arithmetic computations
• Oblivious transfer for conditional data access