Membership Inference Attack

The adversary only observes the final predicted class label from the target model, not confidence scores. This is the most restrictive and realistic threat model.

• Mechanism: Exploits the model's tendency to be more confident and correct on training data. The attack measures robustness to adversarial perturbations; members require larger perturbations to change the predicted label.
• Key Metric: The distance to the decision boundary is smaller for non-members.
• Example: The Label-Only MIA by Choquette-Choo et al. (2021) augments query samples with random noise and counts how many perturbed versions flip the predicted label.

The adversary accesses the model's posterior probabilities (confidence scores) for all classes. This is the classic setting introduced by Shokri et al. (2017).

• Mechanism: Trains a binary attack classifier on the target model's confidence vectors. The attack model learns to distinguish the distribution of confidences for training data (typically higher and sharper) from non-training data.
• Shadow Models: The adversary trains multiple local models on similar data distributions to generate labeled (member/non-member) confidence vectors for training the attack classifier.
• Vulnerability: Models with high confidence on training points and low confidence on unseen points leak the most membership signal.

The adversary has full access to the trained model's weights and architecture. This enables more powerful statistical analyses.

• Mechanism: Computes per-sample loss values or gradient norms using the final model parameters. Training samples typically exhibit lower loss and smaller gradient norms because the model has been explicitly optimized to minimize them.
• Likelihood Ratio Attacks: Carlini et al. (2022) propose the Likelihood Ratio Attack (LiRA), which trains shadow models to estimate the probability density of loss values for members vs. non-members, then performs a hypothesis test on the target sample's loss.
• Advantage: LiRA achieves a near-perfect true positive rate at low false positive rates by calibrating the per-sample hardness.

The adversary compares the target model's output on a query sample against a reference model trained on a disjoint dataset from the same distribution.

• Mechanism: If the target model assigns significantly higher confidence to a sample than the reference model, the sample was likely in the target's training set. This controls for inherent sample difficulty.
• Gap Attack: Yeom et al. (2018) formalize this by thresholding the difference between the target model's loss and a reference model's loss.
• Practicality: Requires the adversary to train a high-quality reference model, which is computationally expensive but provides a strong baseline for membership signal.

The adversary exploits the model's invariance to specific data augmentations applied during training.

• Mechanism: If a model was trained with a specific augmentation strategy (e.g., random cropping, horizontal flips), it will produce more consistent predictions across augmented versions of training samples than non-training samples.
• Augmentation Multiplicity: The adversary queries the model with multiple augmented copies of the target sample and measures the consistency of the output distribution. High consistency implies membership.
• Defense Implication: Models trained without augmentation or with deterministic pipelines are less susceptible, but this conflicts with generalization best practices.

Targets generative language models by analyzing the perplexity or probability assigned to token sequences.

• Mechanism: Training sequences are assigned lower perplexity (higher likelihood) by the model. The attack thresholds the per-token loss averaged over the sequence.
• Canary Extraction: Carlini et al. (2019) demonstrate that language models memorize rare sequences verbatim. By inserting canary strings into the training data, membership is trivially detected if the model can complete the canary with high confidence.
• Exposure Metric: Quantifies how much higher the model ranks the true continuation of a sequence compared to random alternatives.

The gold-standard mathematical framework for limiting membership leakage. DP-SGD clips per-sample gradients and injects calibrated Gaussian noise during training. This ensures the model's final parameters are statistically indistinguishable regardless of any single record's inclusion. The privacy guarantee is controlled by the privacy budget (epsilon); lower values provide stronger protection against membership inference but may reduce model utility on rare tail data.

A defensive architecture where a sensitive 'teacher' model trained on private data is used to label public or synthetic data, which then trains a public 'student' model. Because the student never sees the original private records, membership inference attacks against it are significantly less effective. Strict temperature scaling during distillation further smooths the output probabilities, masking the overconfident predictions on training points that membership inference attacks typically exploit.

Standard regularization methods reduce overfitting, which is the primary source of membership vulnerability. Key techniques include:

• L2 Regularization: Penalizes large weights to prevent memorization of outliers.
• Dropout: Randomly disables neurons during training, forcing the model to learn redundant patterns rather than specific records.
• Early Stopping: Halts training before the model begins to memorize individual data points, keeping the generalization gap narrow.

A min-max game where the model is trained to simultaneously minimize its primary loss while maximizing the error of a simulated membership inference attacker. This directly optimizes the model's parameters to be indistinguishable on training versus non-training points. Gradient reversal layers are commonly used to flip the attacker's gradient, ensuring the feature extractor learns representations that are useful for the task but uninformative for membership status.

Limiting the granularity of information exposed through prediction APIs. Defenses include:

• Top-k Predictions: Returning only the top few class labels instead of the full probability vector.
• Confidence Score Masking: Rounding or clipping confidence values to prevent high-precision score analysis.
• Query Rate Limiting: Blocking the high-volume, systematic queries required to build shadow models for membership inference attacks.

Proactive testing using canary gradients and exposure metrics to quantify how much a model has memorized its training data. By inserting unique, out-of-distribution sequences into the training set and measuring the model's perplexity on them, engineers can empirically estimate membership inference risk before deployment. This allows for dynamic adjustment of privacy parameters based on measured leakage rather than theoretical bounds alone.