Expected Calibration Error (ECE)

ECE partitions predictions into M equally-spaced bins based on confidence scores (e.g., 0-0.1, 0.1-0.2). For each bin:

• Accuracy is the fraction of correctly classified samples.
• Confidence is the mean predicted probability. The absolute difference |acc(B_m) - conf(B_m)| is weighted by the bin's relative size. This prevents a model that is perfectly calibrated on 99% of data but wildly off on 1% from hiding its error.

ECE is visualized through a reliability diagram, which plots accuracy against confidence for each bin. A perfectly calibrated model (ECE=0) follows the identity line y=x. Deviations reveal systematic biases:

• Below the diagonal: Model is overconfident (confidence > accuracy).
• Above the diagonal: Model is underconfident (confidence < accuracy). Modern neural networks typically exhibit a characteristic overconfidence bias, especially on out-of-distribution inputs.

While both measure calibration, they capture distinct failure modes:

• ECE measures the maximum calibration gap and is non-differentiable due to hard bin assignments, making it unsuitable as a direct training loss.
• Brier Score is a strictly proper scoring rule that decomposes into refinement and calibration terms. It is differentiable but can be dominated by discrimination performance.
• Adaptive ECE variants use equal-mass binning to ensure each bin has sufficient samples, addressing ECE's sensitivity to bin count.

A post-hoc calibration method that directly minimizes ECE by learning a single scalar parameter T > 0 applied to the logits before softmax:

• T = 1: Original probabilities unchanged.
• T > 1: Softens probabilities, reducing overconfidence.
• T < 1: Sharpens probabilities, increasing confidence. Temperature scaling is optimized on a held-out validation set using NLL loss and does not affect the model's ranking or accuracy—it only adjusts the confidence magnitudes.

Standard ECE computes calibration only for the predicted class, ignoring miscalibration across all other classes. Classwise ECE extends the metric by computing calibration errors for every class independently and averaging them:

• Detects when a model is systematically overconfident about a specific background class.
• Critical for safety-critical applications where the confidence of a "no-finding" class must be precisely calibrated.
• Requires more samples per bin to maintain statistical reliability.

ECE has known statistical pathologies:

• Fixed bin width can leave many bins empty in low-data regimes, producing unstable estimates.
• Non-differentiability prevents direct optimization via gradient descent.
• Paradox of equal ECE: A model that always predicts 50% confidence with 50% accuracy achieves ECE=0, despite being useless for decision-making. Always pair ECE with a discrimination metric like accuracy or AUC.