Reliability Diagram

The x-axis is constructed by partitioning the model's predicted confidence scores (e.g., 0.0-0.1, 0.1-0.2) into M equally spaced bins. Each prediction is assigned to a bin based on its confidence. The choice of bin number (e.g., M=10) is a hyperparameter; too few bins oversmooth the calibration curve, while too many introduce high variance. Common strategies include equal-width bins or bins with an equal number of samples (equal-frequency binning).

For each bin, two key values are calculated:

• Average Confidence: The mean of the predicted confidence scores for all samples in the bin.
• Empirical Accuracy: The proportion of samples in the bin where the model's predicted class matches the true label. A perfectly calibrated model will have points where Average Confidence = Empirical Accuracy for every bin, resulting in points lying directly on the diagonal y = x line of the plot.

The central element of the diagram is the calibration curve, formed by plotting the average confidence (x-coordinate) against the empirical accuracy (y-coordinate) for each bin. The shape of this curve relative to the diagonal reveals the type of miscalibration:

• Underconfidence: Curve lies above the diagonal (accuracy > confidence).
• Overconfidence: Curve lies below the diagonal (confidence > accuracy).
• Systematic Bias: A consistent offset from the diagonal across all confidence levels.

The diagonal line y = x represents the ideal state of perfect calibration. It serves as the visual benchmark. The distance of the calibration curve from this line provides an immediate, intuitive assessment of miscalibration. A model can have high accuracy but still be poorly calibrated if its curve deviates significantly from the diagonal, indicating its confidence scores are not reliable probability estimates.

Often displayed as a bar chart beneath the main calibration curve, the histogram shows the distribution of predictions across the confidence bins. This reveals where the model's predictions are concentrated. A model that makes most predictions with very high confidence (e.g., bins 0.9-1.0) but is miscalibrated there is particularly problematic, as its most certain predictions are wrong. It highlights the confidence distribution of the classifier.

The diagram provides the visual foundation for scalar calibration error metrics. The Expected Calibration Error (ECE) is directly computed from the binned data shown in the diagram: ECE = Σ (|B_m| / n) * |acc(B_m) - conf(B_m)|, where the sum is over all bins M. The Maximum Calibration Error (MCE) is the maximum observed discrepancy across all bins. The diagram makes these abstract metrics concrete by showing exactly where and how the miscalibration occurs.

Calibration error is the quantitative discrepancy between a model's predicted confidence scores and its actual empirical accuracy. It measures how well a confidence score of X% corresponds to a X% chance of being correct.

• Expected Calibration Error (ECE) is the most common scalar summary, calculated by binning predictions and averaging the absolute difference between average confidence and accuracy per bin.
• Proper scoring rules, like the Brier Score, are loss functions that directly penalize miscalibration during training.

Uncertainty Quantification (UQ) is the field focused on measuring and interpreting the different types of uncertainty in a model's predictions. A reliability diagram primarily visualizes total predictive uncertainty.

• Aleatoric Uncertainty is inherent, irreducible noise in the data (e.g., sensor error).
• Epistemic Uncertainty stems from the model's lack of knowledge, reducible with more data.
• Methods like Monte Carlo Dropout and Deep Ensembles are used to estimate these uncertainties.

Post-hoc calibration refers to techniques applied to a trained model's outputs to improve their probabilistic reliability without retraining. These methods are validated using reliability diagrams.

• Platt Scaling fits a logistic regression model to map raw scores (logits) to calibrated probabilities.
• Temperature Scaling is a simpler, single-parameter variant of Platt scaling that adjusts the softmax distribution's sharpness.
• Isotonic Regression is a non-parametric method that learns a piecewise constant calibration map.

Selective classification, or classification with a rejection option, allows a model to abstain from making a prediction when its confidence is below a threshold. Reliability diagrams inform the setting of this threshold.

• A Risk-Coverage Curve plots the error rate (risk) against the fraction of samples predicted (coverage), defining the accuracy-abstention trade-off.
• This is critical for high-stakes applications where an incorrect but confident prediction is dangerous.

A proper scoring rule is a function that evaluates the quality of a probabilistic forecast, encouraging the forecaster to report their true belief. They are the training objectives that calibration methods aim to optimize.

• Negative Log-Likelihood (NLL) is the standard proper score used for training classification models.
• Brier Score is the mean squared error between the predicted probability vector and the one-hot encoded true label.
• Minimizing these scores during training generally improves, but does not guarantee, calibration.

Conformal prediction is a model-agnostic framework that produces prediction sets (not single labels) with guaranteed statistical coverage. It provides a rigorous, distribution-free alternative to confidence scores.

• It guarantees that the true label will be contained in the prediction set, say, 95% of the time.
• Conformal Quantile Regression extends this to regression tasks.
• Unlike reliability diagrams which assess calibration, conformal prediction enforces it by construction.