Top-Label Calibration

Top-label calibration is defined by the identity P(Ŷ = Y | max P̂ = p) = p. This means that for all predictions where the model's highest confidence is, say, 90%, the model should actually be correct exactly 90% of the time. It is a weaker condition than full multiclass calibration, which requires this to hold for every individual class. This makes it a more pragmatic target for models where only the single predicted label matters, such as in selective classification or decision-making systems.

A model can be perfectly top-label calibrated while being poorly classwise calibrated. For instance, a model might always predict class 'A' with 70% confidence and be correct 70% of the time, satisfying the top-label criterion. However, when it predicts class 'B' with 30% confidence, it might only be correct 10% of the time. Classwise calibration would detect this failure, but top-label calibration ignores it, as it only cares about the probability of the single winning class.

Standard Expected Calibration Error (ECE) can be adapted to specifically measure top-label miscalibration. The Adaptive Calibration Error (ACE) uses an adaptive binning scheme that ensures each bin has an equal number of predictions, focusing the metric on the model's maximum confidence. This avoids the pitfalls of fixed-width bins that can be empty in high-confidence regions, providing a more robust measure of how well the model's top prediction confidence aligns with its accuracy.

Top-label calibration is the natural criterion for selective classification systems, where a model can abstain from making a prediction. The decision to predict or abstain is typically based on a threshold on the maximum predicted probability. If the model is top-label calibrated, this threshold directly translates to a guaranteed accuracy level for the covered predictions. This allows for a precise, quantifiable trade-off between coverage (the fraction of inputs predicted on) and risk (the error rate on those inputs).

Temperature scaling, a single-parameter extension of Platt scaling, is a highly effective post-hoc method for achieving top-label calibration in modern neural networks. It works by dividing the output logits by a scalar temperature T > 1 before applying the softmax function. This softens the probability distribution, reducing overconfidence without altering the rank order of the predictions. The optimal T is found by minimizing the Negative Log-Likelihood (NLL) on a held-out validation set, directly optimizing for top-label calibration.

A critical limitation is that top-label calibration does not guarantee robustness to distribution shift. A model can be perfectly calibrated on its in-distribution test set but become severely miscalibrated on out-of-distribution data. For example, a model might assign 99% confidence to an incorrect prediction on an anomalous input. This is because top-label calibration is a marginal property and does not account for epistemic uncertainty—the model's lack of knowledge about inputs far from its training data.