Confidence Calibration

The primary metric for measuring miscalibration. ECE partitions predictions into M equally-spaced bins based on confidence and computes the weighted average of the difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• A lower ECE indicates better calibration
• Sensitive to binning strategy; adaptive binning is often preferred for imbalanced data

A post-hoc calibration method that uses a single scalar parameter T (temperature) to soften or sharpen the output logits before the softmax function. It is the simplest and most effective Platt scaling variant for neural networks.

• T > 1: Softens probabilities, reducing over-confidence
• T < 1: Sharpens probabilities, reducing under-confidence
• Optimized on a held-out validation set to minimize Negative Log-Likelihood (NLL)
• Does not change the model's accuracy or rank-ordering of predictions

A non-parametric calibration method that learns a piecewise constant, monotonically increasing mapping from raw model scores to calibrated probabilities. It makes no assumptions about the shape of the miscalibration curve.

• More flexible than temperature scaling, but prone to overfitting on small datasets
• Ideal for models with complex, non-sigmoidal distortion in their confidence estimates
• Requires a dedicated calibration dataset of at least 1,000 samples for stable fitting

A strictly proper scoring rule that measures the mean squared difference between the predicted probability and the actual binary outcome. It jointly evaluates discrimination and calibration.

• Formula: BS = (1/N) Σ (p_i - y_i)²
• Ranges from 0 (perfect) to 1 (worst)
• Decomposable into a refinement term and a calibration term, allowing granular error analysis

A standard loss function that also serves as a rigorous calibration metric. NLL heavily penalizes high-confidence misclassifications, making it sensitive to over-confident errors.

• Formula: NLL = - Σ y_i * log(p_i)
• A lower NLL indicates both better accuracy and better calibration
• Often used as the optimization target when learning the temperature parameter for temperature scaling

The primary empirical metric for quantifying miscalibration. ECE partitions predictions into M equally-spaced bins based on confidence and computes a weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• A perfectly calibrated model has an ECE of 0.
• Lower ECE indicates that a model's probability estimates are reliable for downstream decision-making.

A visual diagnostic tool for calibration that plots observed accuracy against predicted confidence. For a perfectly calibrated model, the plot is a diagonal line.

• Under-confidence: The curve sits above the diagonal (accuracy > confidence).
• Over-confidence: The curve sits below the diagonal (confidence > accuracy).
• Used to visually identify specific confidence regions where a model's probability estimates break down.

A post-hoc calibration method that uses a single scalar parameter T > 0 to soften or sharpen the output logits before the softmax function. It is optimized on a held-out validation set to minimize the Negative Log-Likelihood (NLL).

• T > 1: Softens probabilities, reducing over-confidence.
• T < 1: Sharpens probabilities, reducing under-confidence.
• It preserves model accuracy and ranking but fixes the probability magnitudes.

A parametric calibration method that fits a logistic regression model on top of a classifier's raw output scores. It transforms scores into calibrated probabilities using learned scalar parameters A and B.

• Formula: P(y=1|x) = 1 / (1 + exp(A * f(x) + B))
• Works best with Support Vector Machines and binary classifiers.
• Less effective than temperature scaling for modern deep neural networks with many classes.

A non-parametric calibration method that learns a piecewise constant, monotonically increasing function to map uncalibrated scores to calibrated probabilities. It is more flexible than Platt scaling but prone to overfitting on small datasets.

• Does not assume a specific sigmoidal shape.
• Minimizes a squared loss function subject to monotonicity constraints.
• Requires a larger calibration set to avoid learning noise.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and the actual binary outcomes. It decomposes into a refinement term and a calibration term.

• Formula: BS = (1/N) Σ (p_i - o_i)²
• Ranges from 0 (perfect) to 1 (worst).
• Unlike ECE, it is sensitive to both discrimination and calibration, penalizing models that are confident but wrong more heavily.