Memorization

The number of parameters relative to the training dataset size is the single strongest predictor of memorization. Overparameterized models—where parameters vastly exceed training examples—possess sufficient capacity to encode exact training samples within their weights.

• Double descent phenomenon: Memorization increases sharply after the interpolation threshold where models perfectly fit training data
• Large language models with billions of parameters can memorize and later regurgitate verbatim text sequences from their training corpus
• Capacity control through architecture selection directly bounds the maximum information that can be stored per sample

Extended training beyond the point of optimal generalization causes models to transition from learning patterns to memorizing individual examples. Overfitting manifests as a growing divergence between training and test performance.

• Early stopping monitors validation loss and halts training before memorization dominates
• Epoch count directly correlates with membership inference vulnerability—more epochs mean higher attack success rates
• Models trained to near-zero training loss exhibit perfect memorization of atypical or unique training samples
• The privacy-utility tradeoff requires balancing generalization against the risk of encoding individual records

The frequency with which specific examples appear in the training corpus dramatically amplifies memorization. Duplicated sequences receive stronger gradient signals, making them more likely to be encoded verbatim.

• Deduplication of training data is a critical preprocessing step that reduces memorization without sacrificing model quality
• Sequences repeated more than 5-10 times in web-scale datasets show exponentially higher extraction risk
• Canary insertion techniques deliberately place unique sequences to measure and audit memorization rates
• The long tail of rare or unique samples is paradoxically both most vulnerable to memorization and most valuable for privacy attacks

Samples that deviate significantly from the population distribution are memorized more readily than typical examples. Outliers and anomalies create distinctive gradient patterns that the model must encode to minimize loss.

• Gradient norm for atypical examples is substantially larger, creating stronger parameter updates that embed the sample
• Exposure metrics quantify how much more likely a model is to generate a specific canary sequence versus random alternatives
• Minority subgroups and rare attributes in training data face disproportionate memorization risk
• Adversarial examples at the boundary of the training distribution are particularly susceptible to extraction

Techniques that constrain model complexity serve dual purposes: improving generalization and reducing memorization. Regularization limits the effective capacity available for encoding individual samples.

• Weight decay (L2 regularization) penalizes large parameter values, discouraging the model from dedicating parameters to single examples
• Dropout randomly disables neurons during training, preventing co-adaptation that facilitates memorization
• Label smoothing replaces hard one-hot targets with soft distributions, reducing the confidence gap exploited by membership inference
• DP-SGD adds calibrated noise during optimization, providing formal privacy guarantees that directly bound memorization

The distribution of model confidence scores reveals memorization patterns. Overconfident predictions on training samples versus higher uncertainty on test samples create the signal exploited by membership inference attacks.

• Prediction entropy is systematically lower for memorized training examples
• Temperature scaling recalibrates confidence scores to reduce the gap between training and non-training distributions
• Selective classification with abstention thresholds can mask the confidence signals that attackers rely on
• Well-calibrated models exhibit smaller memorization-induced confidence discrepancies, naturally resisting inference attacks

A mathematical framework that provides provable privacy guarantees by injecting calibrated noise into computations, ensuring the output is statistically indistinguishable whether or not any single individual's data is included. DP directly bounds memorization by limiting the influence any single training record can have on the final model.

• Epsilon (ε) quantifies the privacy loss budget — smaller values mean stronger guarantees
• Delta (δ) represents the probability of catastrophic privacy failure
• Composes gracefully across multiple queries or training steps

Differentially Private Stochastic Gradient Descent is the primary training algorithm that operationalizes differential privacy for deep learning. It prevents memorization at the gradient level by applying two critical operations:

• Per-sample gradient clipping bounds the L2 norm of each individual gradient, limiting the maximum influence of any single record
• Gaussian noise addition injects calibrated random noise into the aggregated gradient before the weight update
• The clipping threshold and noise multiplier jointly determine the privacy budget (ε)

A modeling error where a machine learning algorithm learns the noise and random fluctuations in the training data to the extent that it negatively impacts generalization. Overfitting is the primary amplifier of memorization — models that overfit exhibit large confidence gaps between training and test samples, creating the signal that membership inference attacks exploit.

• Early stopping halts training before overfitting sets in
• Weight decay penalizes large parameter values that enable memorization
• Dropout randomly disables neurons during training to prevent co-adaptation to specific examples

A measure of uncertainty in a model's output probability distribution, calculated as the negative sum of probabilities multiplied by their logarithms. Low entropy on training samples versus higher entropy on non-training samples serves as the primary signal exploited by membership inference attacks.

• Training samples typically produce peaked, high-confidence distributions
• Non-training samples yield flatter, more uncertain distributions
• Entropy-based defenses include temperature scaling and selective classification to equalize this gap

The process of removing the influence of specific training data points from a trained model without full retraining. Exact unlearning eliminates memorization of deleted records entirely, while approximate unlearning reduces it below a statistical threshold.

• SISA Training partitions data into independent shards to limit retraining scope
• Gradient-based unlearning applies reverse updates to scrub specific influences
• Enables compliance with data deletion requests under regulations like GDPR's right to be forgotten