Calibration

A model is perfectly calibrated if, for all samples where the predicted probability is P, the empirical accuracy is exactly P. For example, among 100 predictions assigned 80% confidence, exactly 80 should be correct. This alignment eliminates the confidence gap between training and non-training samples that membership inference attacks exploit. Formally, P(ŷ = y | p̂ = p) = p, ∀ p ∈ [0,1].

ECE is the primary metric for measuring miscalibration. It partitions predictions into M equal-width confidence bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Calculation: ECE = Σ (|B_m|/n) × |acc(B_m) - conf(B_m)|
• Interpretation: Lower ECE indicates better calibration
• Typical threshold: ECE < 0.05 is considered well-calibrated
• Limitation: ECE is sensitive to binning strategy and can be gamed with wide bins

A reliability diagram (or calibration curve) visually plots observed accuracy against predicted confidence. A perfectly calibrated model follows the diagonal identity line.

• Above diagonal: Model is underconfident — accuracy exceeds predicted probability
• Below diagonal: Model is overconfident — predicted probability exceeds accuracy
• Gap analysis: The area between the curve and diagonal quantifies miscalibration
• Histogram overlay: Often includes a bar chart showing sample density per confidence bin to highlight where calibration matters most

Temperature scaling is the most widely used post-hoc calibration method. It applies a single scalar parameter T > 0 to divide all logits before the softmax, smoothing or sharpening the output distribution without affecting accuracy or rank order.

• T = 1: Original probabilities unchanged
• T > 1: Softens probabilities, reducing overconfidence
• T < 1: Sharpens probabilities, increasing confidence
• Optimization: T is learned by minimizing NLL on a held-out validation set
• Key advantage: Simple, effective, and preserves model accuracy

The confidence gap is the difference between a model's prediction confidence on training versus non-training samples. Membership inference attacks exploit this gap: training samples typically receive higher confidence scores.

• Well-calibrated models exhibit smaller confidence gaps, reducing attack surface
• Overconfident memorization of training data widens the gap
• Temperature scaling can narrow the gap by softening overconfident training predictions
• Selective classification can mask the gap by abstaining on uncertain inputs
• Adversarial regularization explicitly minimizes this gap during training

Calibration behaves differently across prediction tasks. Binary calibration measures whether predicted probabilities for the positive class match observed frequencies. Multi-class calibration requires that confidence is calibrated for every class simultaneously.

• Class-wise ECE: Computes ECE independently per class, then averages
• Top-label calibration: Only evaluates calibration of the predicted class
• Marginal calibration: Ensures per-class probabilities are calibrated across all samples
• Confidence calibration: The most common variant — only the maximum predicted probability is evaluated
• Multi-class models often appear well-calibrated on top-label metrics while exhibiting severe per-class miscalibration

Overfitting occurs when a model learns the statistical noise in training data rather than the underlying signal, leading to poor generalization.

• Memorization: An extreme form of overfitting where the model encodes exact or near-exact copies of individual training examples in its parameters.
• Calibration Link: Overfit models exhibit extreme overconfidence on training points and high uncertainty on unseen data, creating a wide calibration gap.
• MIA Exploit: This confidence disparity is the fundamental vulnerability that membership inference attacks exploit to determine if a record was in the training set.

A per-instance metric that quantifies the likelihood a specific training record can be successfully identified by a membership inference attack.

• Calculation: Often derived from the model's loss, confidence, or entropy for that specific record, compared against a reference distribution of non-member records.
• Application: Enables targeted protective measures—records with high risk scores can be prioritized for unlearning, stronger differential privacy noise, or exclusion from future training.
• Calibration Dependency: The accuracy of a privacy risk score is directly dependent on how well-calibrated the underlying model's confidence estimates are.