Training Data Extraction

The root cause of extraction attacks. Large models, particularly language models, can memorize rare or duplicated sequences from their training data. Verbatim memorization occurs when a model stores an exact copy of a string rather than learning a generalizable pattern. Attackers exploit this by prompting the model in ways that lower the probability threshold for generating memorized content, effectively turning the model into a search engine for its training set. This is distinct from semantic memorization, where only the gist is retained.

A primary technique for extraction involves forcing the model into a low-probability generation mode. By prompting with a rare or nonsensical prefix, the attacker causes the model to diverge from high-likelihood, generic text. In this unstable state, the model relies heavily on its memorized priors, often completing the sequence with verbatim text from a single training document. This method systematically bypasses the model's default behavior of generating plausible, generic completions.

To measure vulnerability, researchers use the exposure metric. This involves inserting known 'canary' strings into the training data and then calculating the log-perplexity of those strings under the trained model relative to a reference model. A high exposure score indicates that the model assigns an anomalously high probability to the secret, quantifying the risk of extraction. This metric is critical for auditing models before deployment.

In computer vision, extraction manifests as model inversion. An attacker starts with random noise and optimizes it to maximize the confidence score for a target class. Due to the model's learned feature representations, the resulting image often contains recognizable, photorealistic details of training subjects. This reconstructs private visual data, such as faces or medical scans, directly from the model's weights without any access to the original images.

A practical attack vector involves prompt engineering to elicit memorized data. Attackers use strategies like:

• Prefix injection: Providing the start of a known document.
• Repetition: Forcing the model to repeat a token many times until it falls into a memorized loop.
• Template filling: Asking the model to complete structured data like email addresses or code blocks. These techniques exploit the model's autoregressive nature to sequentially reveal stored information.

A precursor to extraction that determines whether a specific record was in the training set. Black-box attacks analyze confidence scores, while white-box attacks exploit gradients. The adversary trains shadow models on synthetic data to build a binary classifier that distinguishes members from non-members. Key signals include:

• Higher confidence on true labels for training data
• Larger gaps between top-1 and top-2 predictions
• Lower loss values on memorized examples

Goes beyond membership inference to reconstruct representative features of a class. An attacker queries the model and optimizes an input to maximize the confidence score for a target class, revealing a prototype of what the model associates with that label. In facial recognition systems, this can produce recognizable approximations of individuals' faces from the training data.

The mathematical gold standard for defending against extraction attacks. Provides provable guarantees that the output of a computation reveals no information about any single individual. Implemented through:

• Gaussian Mechanism: Adds noise calibrated to L2 sensitivity
• Laplace Mechanism: Adds noise calibrated to L1 sensitivity
• Privacy Budget (ε): Quantifies total allowable privacy loss

Differentially Private Stochastic Gradient Descent modifies standard training to bound individual influence. Two critical operations:

• Per-sample gradient clipping: Limits the L2 norm of each example's gradient
• Gaussian noise addition: Adds calibrated noise to the aggregated gradient Combined with subsampling amplification, where random batch selection provides stronger privacy than processing the full dataset.

Quantitative measures of how much a model has memorized specific secrets. Canary insertion places unique random strings in training data, then uses likelihood ratio tests to detect if the model has memorized them. The exposure metric compares a secret's likelihood under the model versus a reference distribution. High memorization scores directly correlate with extraction vulnerability.

The systematic tracking of cumulative privacy loss across multiple operations. Moments Accountant tracks the moment-generating function of the privacy loss random variable for tight bounds. Rényi Differential Privacy uses Rényi divergence for more precise composition analysis. Essential for ensuring the total privacy expenditure stays within the allocated epsilon budget throughout training.