Expected Calibration Error (ECE)

ECE partitions predictions into M equally-spaced confidence bins (e.g., 0-10%, 10-20%). For each bin, it computes the absolute difference between the average confidence and the observed accuracy within that bin. The final ECE is a weighted average of these gaps, where weights correspond to the proportion of samples in each bin. This discretization makes the metric computationally tractable and visually interpretable via reliability diagrams.

Formally, ECE is defined as:

ECE = Σ (|B_m| / n) * |acc(B_m) - conf(B_m)|

Where:

• n is the total number of samples
• B_m is the set of samples in bin m
• acc(B_m) is the fraction of correctly classified samples in bin m
• conf(B_m) is the average predicted probability in bin m

A perfectly calibrated model has an ECE of 0.

ECE is almost always accompanied by a reliability diagram. This plot maps average confidence on the x-axis against observed accuracy on the y-axis. A perfectly calibrated model follows the identity line (y=x) . Any deviation above the line indicates underconfidence, while deviation below the line indicates overconfidence—a common pathology in modern deep neural networks.

Despite its popularity, ECE has known weaknesses:

• Bin Sensitivity: The choice of bin count (M) is arbitrary and significantly impacts the score. Too few bins mask miscalibration; too many create noisy, empty bins.
• Class Imbalance Blindness: ECE can be misleadingly low on highly imbalanced datasets where a model predicts the majority class with high confidence.
• Top-Label Focus: Standard ECE only evaluates the confidence of the single predicted class, ignoring the calibration of the full probability vector.

To address fixed-width binning issues, Adaptive ECE (AdaECE) dynamically sizes bins to ensure each contains an equal number of samples. Another refinement, Classwise ECE, computes the calibration error independently for each class and averages the results, preventing a well-calibrated dominant class from hiding severe miscalibration in minority classes. These variants provide a more robust picture of model trustworthiness.

ECE is a measure of marginal calibration and is not a strictly proper scoring rule. This means minimizing ECE to zero does not guarantee a perfect model; a constant predictor outputting the base rate can achieve zero ECE. For a more complete evaluation, ECE should be paired with a proper scoring rule like the Brier Score or Negative Log-Likelihood (NLL) , which jointly assess calibration and discrimination.

The process of aligning a model's predicted probability of correctness with the actual empirical frequency of being correct. A perfectly calibrated model that predicts 70% confidence will be correct exactly 70% of the time. Expected Calibration Error (ECE) is the primary scalar metric used to quantify miscalibration by measuring the gap between confidence and accuracy across binned predictions. Post-hoc methods like temperature scaling and Platt scaling are commonly used to repair calibration without retraining.

A visual diagnostic tool that plots expected sample accuracy against confidence to reveal calibration errors. Predictions are partitioned into M equal-width bins, and the average confidence within each bin is plotted against the observed accuracy. A perfectly calibrated model follows the identity line (y=x). The ECE is computed as the weighted average of the absolute differences between each bin's confidence and accuracy, making the reliability diagram the graphical counterpart to the ECE scalar.

A strictly proper scoring rule that measures the mean squared difference between the predicted probability and the actual binary outcome. Unlike ECE, which only assesses calibration, the Brier Score jointly evaluates both calibration and discrimination (refinement). It decomposes mathematically into a calibration loss term and a refinement term, providing a more complete picture of probabilistic forecast quality. A lower Brier Score indicates better overall performance.

A post-hoc calibration method that divides the logits by a single scalar parameter T (temperature) to soften the softmax output without changing the model's accuracy or prediction ranking. When T > 1, the output distribution becomes smoother and less overconfident. The optimal T is learned by minimizing the Negative Log-Likelihood (NLL) on a held-out validation set. This single-parameter fix is the most common method for reducing ECE in modern neural networks.

Model uncertainty arising from a lack of knowledge or data, which can theoretically be reduced with more training samples or a better model architecture. While ECE measures calibration error, epistemic uncertainty quantifies the model's ignorance. High epistemic uncertainty often correlates with miscalibration in out-of-distribution regions. Methods like Deep Ensembles and Monte Carlo Dropout estimate this uncertainty by measuring prediction variance across multiple stochastic forward passes.

A related metric that measures the worst-case deviation between confidence and accuracy across all bins, rather than the weighted average computed by ECE. While ECE provides a global summary, MCE identifies the single most miscalibrated bin, which is critical for safety-sensitive applications where any large miscalibration is unacceptable. In high-stakes domains like medical diagnosis or autonomous driving, minimizing MCE is often prioritized over minimizing ECE.