Reliability Diagram

The diagonal line (y = x) represents perfect calibration. A model is perfectly calibrated if, for all predictions with a confidence of P%, the actual accuracy is exactly P%.

• Above the diagonal: The model is underconfident—its accuracy is higher than its predicted probability.
• Below the diagonal: The model is overconfident—its predicted probability exceeds its actual accuracy.
• Deviation magnitude indicates the severity of miscalibration at that confidence level.

The diagram partitions the probability interval [0,1] into M equal-width or equal-mass bins. For each bin:

• X-axis: The mean predicted probability of samples in that bin.
• Y-axis: The empirical frequency of positive outcomes.
• Bin count (M) is a hyperparameter; 10–20 bins are typical. Too few bins obscure detail, while too many create noisy, unreliable estimates.
• Equal-mass binning (each bin contains the same number of samples) ensures consistent statistical reliability across the plot.

A reliability diagram is often paired with a marginal histogram showing the distribution of predicted probabilities.

• Sharpness: A model concentrated near 0 and 1 is sharp (confident). A model spread across the range is diffuse.
• Sharpness is distinct from calibration. A model can be perfectly calibrated but uselessly diffuse (e.g., always predicting the base rate).
• The histogram reveals where the model operates—critical for understanding if miscalibration occurs in regions with sufficient data density.

The Expected Calibration Error (ECE) is the primary scalar summary derived from a reliability diagram.

• Formula: ECE = Σ (|B_m| / N) × |acc(B_m) − conf(B_m)|, where B_m is bin m, N is total samples, acc is accuracy, and conf is mean confidence.
• ECE is the weighted average of the gaps between each bin's point and the diagonal.
• Limitation: ECE is sensitive to binning choices and can be gamed by models that distribute errors uniformly. Always inspect the full diagram alongside the scalar.

For multiclass classification, reliability diagrams can be constructed in several ways:

• Top-label calibration: Plots the confidence of the predicted class against the accuracy of that prediction. Most common for classifiers with many classes.
• Classwise calibration: Generates a separate diagram for each class, plotting P(class=k) against the observed frequency of class k. Stricter and more diagnostic.
• Confidence histograms for multiclass problems often reveal whether a model collapses probability mass onto a single class regardless of uncertainty.

The shape of the reliability curve directly informs the choice of recalibration method:

• Sigmoidal S-shape (overconfident at extremes, underconfident in the middle): Platt scaling or temperature scaling are effective parametric fixes.
• Non-monotonic or irregular deviations: Isotonic regression is preferred as it makes no functional form assumptions.
• Systematic underconfidence across all bins: Indicates a need for label smoothing during training or adjusting the decision threshold.
• Always recalibrate on a held-out validation set to avoid overfitting the calibration map.