Training Data Extraction

Large language models and diffusion models can encode exact copies of training data within their weights. This occurs when rare or unique sequences are encountered repeatedly during training. The model treats these sequences as high-value patterns rather than generalizable features.

• Example: GPT-2 was shown to memorize and output specific names, phone numbers, and addresses when prompted with rare context prefixes.
• Mechanism: The perplexity of memorized sequences is anomalously low compared to generated text, making them statistically detectable.
• Risk factor: Sequences appearing fewer than 100 times in the training corpus are disproportionately vulnerable to memorization.

A sequence is considered k-eidetic if it can be extracted from a model when prompted with a prefix of length k. This formalizes the relationship between prompt length and extraction success.

• Definition: A string s is k-eidetic if an adversary with knowledge of the first k tokens can reliably generate the remaining tokens.
• Implication: Even partial knowledge of a training document (e.g., a standard legal clause or boilerplate header) can enable full extraction.
• Measurement: Researchers quantify memorization by measuring the ratio of extractable sequences across varying prefix lengths, establishing a direct privacy-utility trade-off curve.

The exposure metric measures how much more likely a model is to generate a specific training sequence compared to random chance. This provides a quantitative framework for auditing memorization.

• Formula: Exposure = log₂(Perplexity of random sequence) - log₂(Perplexity of target sequence)
• Thresholds: Exposure values above 40 indicate near-certain memorization; values below 20 suggest minimal risk.
• Application: This metric enables privacy auditors to rank training sequences by extraction risk and apply targeted defenses like differential privacy or data deduplication.

Removing duplicate training examples is one of the most effective defenses against extraction attacks. Models are far less likely to memorize sequences that appear only once.

• Impact: Deduplicating the training corpus reduced extractable memorization by approximately 10x in controlled experiments.
• Technique: Locality-sensitive hashing (LSH) identifies near-duplicate documents across massive datasets without pairwise comparison.
• Limitation: Deduplication does not eliminate memorization entirely; rare but unique sequences (e.g., personal identifiers) remain vulnerable and require additional protections like DP-SGD.

An attack that reconstructs sensitive training data or statistical features by exploiting a model's confidence scores or internal representations. Unlike extraction, which recovers verbatim sequences, inversion aims to generate a representative prototype of a class (e.g., a face image that maximizes the 'Person X' classifier output). Attackers iteratively optimize an input to maximize the model's belief in a target label, effectively reversing the learned mapping.

An attack that determines whether a specific data record was present in a model's training dataset by analyzing its output behavior. Attackers exploit the fact that models often exhibit higher confidence on training samples than on unseen data. Key techniques include:

• Shadow model training to mimic target model behavior
• Loss threshold analysis on prediction vectors
• Likelihood ratio tests comparing output distributions This is often a precursor to extraction attacks, confirming data presence before attempting recovery.

The phenomenon where a neural network encodes exact copies of training data within its weights, making it possible for attackers to extract rare or unique sequences. Large language models are particularly susceptible, memorizing:

• Code snippets with specific variable names
• Personal identifiable information (PII)
• Canonical texts repeated in the corpus Memorization scales with model size, data duplication frequency, and sequence rarity. It is the root cause vulnerability that training data extraction exploits.

A mathematical framework that provides provable privacy guarantees by injecting calibrated statistical noise into data or model outputs. The parameter epsilon (ε) quantifies the privacy loss budget—lower values mean stronger privacy. Key implementations include:

• DP-SGD: Clips per-sample gradients and adds Gaussian noise during training
• PATE: Transfers knowledge via noisy voting from an ensemble of teacher models Differential privacy is the gold-standard defense against both membership inference and training data extraction.

A defense mechanism that truncates or suppresses the full prediction vector returned by an API to prevent attackers from exploiting fine-grained confidence values. Strategies include:

• Returning only the top-k predicted classes
• Rounding or quantizing probability scores
• Replacing softmax outputs with hard labels only This directly reduces the information leakage available to both inversion and membership inference attacks by limiting the attack surface of the model's output layer.

A security process that logs and analyzes incoming inference requests to detect and block suspicious query patterns indicative of extraction or inversion attacks. Auditing systems monitor for:

• High-frequency, semantically similar queries
• Systematic probing of the input space
• Gradient-based optimization patterns in query sequences Effective auditing acts as a runtime anomaly detection layer, terminating sessions that exhibit extraction behavior before meaningful data can be recovered.