Prediction Entropy

Prediction entropy measures the spread of a model's output probabilities. The core vulnerability exploited by membership inference attacks is the entropy gap: training samples typically produce lower-entropy (more confident) predictions than non-training samples.

• Low entropy on a sample suggests it was memorized during training
• High entropy indicates the model is uncertain, typical of unseen data
• Attackers threshold this difference to classify members vs. non-members

For a classification model outputting probability vector p over k classes, prediction entropy H(p) is calculated as:

H(p) = -Σ p_i · log(p_i)

• Maximum entropy occurs with uniform distribution: H = log(k)
• Minimum entropy (0) occurs when one class has probability 1.0
• Top-1 confidence is a related but less informative metric that ignores the distribution of remaining probability mass

Model calibration directly impacts the reliability of entropy as a privacy signal. A miscalibrated model may exhibit systematically overconfident predictions, artificially compressing entropy values.

• Temperature scaling smooths output probabilities, increasing entropy uniformly
• Well-calibrated models show smaller entropy gaps between train and test data
• Expected Calibration Error (ECE) quantifies miscalibration that can mask or amplify membership signals

Defenders can manipulate prediction entropy to thwart membership inference without significantly degrading utility:

• Entropy thresholding: refuse predictions when entropy falls below a calibrated floor
• Selective classification: abstain on low-entropy inputs that signal memorization
• Confidence masking: return only the predicted label, not the full probability vector
• Adversarial regularization explicitly penalizes entropy gaps between training and holdout distributions during training

DP-SGD training with gradient clipping and Gaussian noise injection directly increases prediction entropy by preventing the model from converging to sharp minima on individual training points.

• Higher privacy budgets (ε) permit lower training entropy
• Tighter privacy guarantees force higher entropy across all predictions
• The entropy distribution of a DP-trained model more closely resembles a Bayesian posterior with higher epistemic uncertainty

In label-only attacks, where the model API returns only the predicted class, entropy must be inferred indirectly:

• Attackers query with adversarial perturbations to probe the decision boundary
• The robustness radius—how much perturbation is needed to flip a label—correlates inversely with prediction entropy
• Conformal prediction sets provide guaranteed coverage and can mask the entropy signal by returning multiple plausible labels