Expected Calibration Error (ECE)

ECE operates by sorting all predictions into M equally-spaced confidence bins (e.g., 0-10%, 10-20%). For each bin, it calculates the absolute difference between the average confidence and the observed accuracy. The final ECE is the weighted average of these gaps, where the weight is the proportion of samples in each bin. A perfectly calibrated model has an ECE of 0.

ECE is most often visualized through a reliability diagram, which plots accuracy against confidence. Key features:

• Perfect calibration appears as a diagonal line (y=x)
• Overconfidence manifests as points below the diagonal (high confidence, low accuracy)
• Underconfidence shows points above the diagonal
• The ECE value is the weighted area between the plotted curve and the diagonal

ECE is formally defined as: ECE = Σ (|B_m| / n) * |acc(B_m) - conf(B_m)|

Where:

• B_m is the set of samples in bin m
• n is the total number of samples
• acc(B_m) is the empirical accuracy in bin m
• conf(B_m) is the average predicted probability in bin m

This formulation ensures that bins with more samples contribute proportionally more to the final error score.

Despite its widespread use, ECE has known weaknesses:

• Fixed binning bias: The choice of bin count (typically 10 or 15) is arbitrary and affects the score
• Class imbalance blindness: ECE can be misleadingly low on highly imbalanced datasets where a model predicts the majority class with high confidence
• No sharpness measure: ECE only measures calibration, not the usefulness of the probability estimates (a model predicting the base rate for every sample has perfect ECE but zero utility)
• Adaptive variants like AdaECE and Classwise ECE have been proposed to address these issues

ECE is a measure of calibration error, not a proper scoring rule. It should be used alongside metrics like:

• Brier Score: Decomposes into calibration and refinement components
• Negative Log-Likelihood (NLL): A strictly proper scoring rule that penalizes both poor calibration and poor discrimination
• Expected Normalized Calibration Error (ENCE): A regression-specific adaptation

While ECE is intuitive and widely reported, proper scoring rules provide a more theoretically grounded evaluation of probabilistic predictions.

The most effective post-hoc calibration method for reducing ECE in neural networks is temperature scaling, a single-parameter variant of Platt scaling. It works by:

• Dividing all logits by a scalar temperature T before the softmax
• T > 1 softens probabilities, reducing overconfidence
• T < 1 sharpens probabilities, reducing underconfidence
• The optimal T is found by minimizing NLL on a held-out validation set

Crucially, temperature scaling does not affect the model's accuracy or rank-ordering of predictions—it only recalibrates the confidence scores.

A quantitative metric, typically a probability between 0 and 1, assigned by a model to indicate the likelihood that its prediction is correct. In classification, this is often the softmax output. For generative models, it may be a token-level probability. ECE directly measures how well these scores align with actual empirical accuracy—a model with perfect calibration will have a confidence of 0.8 and be correct exactly 80% of the time for all predictions assigned that score.

A post-hoc calibration method that applies a single scalar parameter T to soften or sharpen the output logits before the softmax function. When T > 1, the distribution flattens, reducing overconfidence. When T < 1, it sharpens, correcting underconfidence. The optimal T is learned on a held-out validation set by minimizing the Negative Log-Likelihood (NLL) . Temperature scaling is the simplest and most widely-used approach to reduce ECE without affecting model accuracy.

The standard visualization for ECE. Predictions are partitioned into M equal-width bins (typically 10) based on confidence. For each bin, the mean confidence is plotted against the observed accuracy. A perfectly calibrated model produces a diagonal line (y = x). Deviations above the diagonal indicate underconfidence; deviations below indicate overconfidence. The ECE is the weighted average of these bin-wise gaps.

ECE primarily captures epistemic uncertainty—the reducible error from limited data or model capacity. This is distinct from aleatoric uncertainty, which is the irreducible noise inherent in the data itself (e.g., overlapping classes). A model can have low ECE while still having high aleatoric uncertainty. Advanced calibration techniques decompose these two sources to provide more nuanced confidence estimates.

A distribution-free framework that provides rigorous mathematical guarantees on coverage. Instead of a single confidence score, it outputs a prediction set containing the true label with a user-specified probability (e.g., 95%). Unlike ECE, which measures average calibration, conformal prediction offers instance-level validity. It serves as a complementary approach when the goal is guaranteed error control rather than probabilistic calibration.

The degradation of a model's calibration over time as the data distribution shifts in production. A model with excellent ECE at deployment may become severely miscalibrated as user behavior, seasonality, or real-world conditions change. Monitoring ECE on live inference data is a critical MLOps practice. Drift detection triggers retraining or recalibration of the temperature parameter to restore statistical alignment between confidence and accuracy.