Classwise Calibration

For a model to be classwise calibrated, the condition P(Y=k | p_k) = p_k must hold for all K classes simultaneously. This means if you collect all predictions where the model assigns a 70% probability to 'cat', exactly 70% of those instances must actually be cats. This granular guarantee prevents a model from being 'calibrated on average' while hiding severe miscalibration in specific minority classes.

Confidence calibration (or top-label calibration) only requires the maximum predicted probability to match accuracy—e.g., when the model is 90% confident in its top prediction, it should be correct 90% of the time. Classwise calibration is strictly stronger: a model can pass confidence calibration while systematically overestimating the probability of a specific class. Classwise calibration implies confidence calibration, but the reverse is not true.

The standard metric is Classwise Expected Calibration Error (CW-ECE), which computes the calibration error independently for each class and averages the results:

• For each class k, partition predictions into M bins by predicted probability
• Compute the weighted absolute difference between average confidence and observed frequency per bin
• Average across all K classes This metric exposes per-class miscalibration that global ECE masks.

Multiclass calibration requires that the entire predicted probability vector matches the empirical class distribution conditioned on that vector. Classwise calibration is a weaker but more practical condition—it only requires marginal calibration per class rather than joint calibration over the full simplex. In practice, classwise calibration is more tractable to measure and optimize while still providing strong per-class reliability guarantees.

Standard temperature scaling with a single scalar parameter cannot fix classwise miscalibration because it applies the same transform to all logits. To achieve classwise calibration, practitioners use:

• Matrix scaling: learning a full matrix W and bias vector b to transform logits class-specifically
• Vector scaling: a diagonal matrix variant that applies per-class temperature parameters
• Dirichlet calibration: fitting a Dirichlet distribution family to the log-probability simplex These methods provide the per-class flexibility needed.

A common classwise calibration failure occurs in imbalanced datasets. A model may achieve low global ECE while drastically overestimating probabilities for rare classes. For example, in a medical diagnosis task with 99% negative cases, the model might assign 60% probability to the disease class when true prevalence at that confidence level is only 20%. Classwise ECE explicitly penalizes this dangerous miscalibration.