Calibration

Also known as calibration plots, these are the primary visual diagnostic tool. Predictions are bucketed into intervals (e.g., 0-10%, 10-20%), and the mean predicted probability is plotted against the true empirical frequency for each bucket. A perfectly calibrated model traces the diagonal y=x line. Deviations above the diagonal indicate underconfidence, while deviations below indicate overconfidence.

The definitive scalar metric summarizing miscalibration. ECE computes the weighted average of the absolute difference between accuracy and confidence across all prediction bins.

• Formula: ECE = Σ (|B_m| / n) * |acc(B_m) - conf(B_m)|
• Interpretation: A lower ECE is better. An ECE of 0.05 means the model's probability estimates are off by 5% on average.
• Limitation: ECE is sensitive to binning strategy; a fixed number of bins (e.g., 10) is standard.

A post-hoc calibration method that fits a logistic regression model on the raw outputs (logits) of a classifier against the true labels. It learns scalar parameters A and B to transform logits into calibrated probabilities: P(y=1|x) = 1 / (1 + exp(A * logit + B)). Platt scaling is highly effective for binary classification and is trained on a held-out validation set to avoid overfitting the correction.

A non-parametric calibration method that learns a piecewise constant, monotonically increasing function to map model scores to empirical probabilities. Unlike Platt scaling, it makes no assumption about the functional form of the distortion.

• Advantage: More flexible; can correct any monotonic miscalibration.
• Disadvantage: Prone to overfitting with small datasets, as it learns a step function directly from the data.

A simple yet highly effective calibration method for multi-class neural networks. A single scalar parameter T (temperature) divides all logits before the softmax function: q_i = exp(z_i / T) / Σ exp(z_j / T). A T > 1 'softens' the probabilities, reducing overconfidence. Optimized on NLL on a validation set, it preserves accuracy while dramatically improving ECE.

A critical distinction. Sharpness refers to the concentration of predictions near 0 or 1, reflecting decisiveness. Calibration is the statistical correctness of those probabilities. A model can be perfectly calibrated but uselessly unsharp (e.g., always predicting the base rate of 5% CTR). Conversely, a sharp but miscalibrated model is dangerously overconfident. The goal is to optimize for both.

The canonical loss function for training calibrated probabilistic classifiers. Log loss heavily penalizes confident but incorrect predictions—a model predicting 99% click probability for a non-click incurs a far greater penalty than predicting 60%.

• Mechanism: Minimizes the negative log-likelihood of the true labels given the predicted probabilities.
• Calibration Connection: A low log loss score implicitly requires good calibration, as systematically overconfident or underconfident predictions inflate the metric.
• Formula: - (y * log(p) + (1-y) * log(1-p)) where p is the predicted probability and y is the true binary label.

The primary visual diagnostic tool for assessing calibration quality. It plots the observed empirical frequency against the predicted probability, bucketed into bins (e.g., 0-10%, 10-20%).

• Perfect Calibration: All points lie exactly on the y = x diagonal.
• Overconfidence: The plotted curve lies below the diagonal—the model predicts higher probabilities than the true occurrence rate.
• Underconfidence: The curve sits above the diagonal—the model is too conservative.
• Sharpness: A separate concept from calibration; a model can be perfectly calibrated but useless if all predictions cluster around the global average (e.g., always predicting 5% CTR).

A scalar summary statistic that quantifies the weighted average absolute difference between accuracy and confidence across all probability bins. ECE reduces the reliability diagram to a single, comparable number.

• Calculation: Partitions predictions into M equally-spaced bins. For each bin m, computes |accuracy(B_m) - confidence(B_m)|, weighted by the proportion of samples in that bin.
• Limitation: ECE is sensitive to binning strategy and does not distinguish between overconfidence and underconfidence within a single value.
• Target: An ECE approaching 0.0 indicates near-perfect calibration.

A post-hoc calibration technique that fits a logistic regression model on top of the raw output scores of a pre-trained classifier. It learns two scalar parameters, A and B, to map uncalibrated scores z to calibrated probabilities: p_calibrated = sigmoid(A * z + B).

• Training Data: Requires a held-out validation set distinct from the original training data to avoid overfitting.
• Best For: Correcting models where the miscalibration pattern is a smooth, sigmoidal distortion of the raw logits.
• Limitation: Assumes the miscalibration function is logistic; fails for more complex, non-monotonic distortions.

A more flexible, non-parametric calibration method that learns a monotonically increasing step function to map raw scores to calibrated probabilities. Unlike Platt Scaling, it makes no assumption about the functional form of the miscalibration.

• Mechanism: Solves a constrained optimization problem to minimize squared error while enforcing monotonicity.
• Risk of Overfitting: The high flexibility can fit noise in small validation sets. A common mitigation is to use cross-validation to generate out-of-fold predictions for calibration training.
• Use Case: Preferred when the reliability diagram shows a jagged, non-sigmoidal deviation from the diagonal.

A simple yet highly effective calibration method for neural networks, particularly multi-class classifiers. It divides all output logits by a single learned scalar parameter T (temperature) before applying the softmax function.

• Effect: T > 1 softens the probability distribution, correcting overconfidence. T < 1 sharpens it, correcting underconfidence.
• Key Advantage: Does not alter the ranking of predictions (AUC remains unchanged), only the confidence scores.
• Optimization: The optimal T is found by minimizing log loss on a held-out validation set.