Logit Calibration

A logit is the raw, unnormalized score output by a neural network's final layer before a softmax function. Logit calibration applies a transformation to these scores so the resulting confidence value—e.g., 0.8—truly means the prediction is correct 80% of the time. An uncalibrated model can be systematically overconfident or underconfident, making its probabilities unreliable for downstream decision-making.

Expected Calibration Error (ECE) is the primary empirical metric for diagnosing miscalibration. The process:

1. Bin all predictions by their confidence score (e.g., [0.0-0.1], [0.1-0.2]).
2. For each bin, compute the difference between average confidence and actual accuracy.
3. ECE is the weighted average of these gaps. A perfectly calibrated model has an ECE of 0. Modern deep networks often exhibit high ECE despite high accuracy.

Platt scaling fits a logistic regression model on top of the raw logits. It learns two parameters, A and B, to output calibrated probabilities: P(y=1|x) = 1 / (1 + exp(A * logit + B)). It is particularly effective for binary classification and Support Vector Machines but can underperform temperature scaling on complex multi-class problems because it does not preserve the relative ordering of all class scores.

Isotonic regression is a non-parametric method that learns a piecewise constant, monotonically increasing function to map uncalibrated scores to calibrated probabilities. Its key advantage is flexibility—it makes no assumptions about the shape of the miscalibration curve. The trade-off is a risk of overfitting on small datasets, making it best suited for large validation sets where the shape of distortion is complex and unknown.

A model can be highly accurate yet poorly calibrated. For example, a 90% accurate model that always reports 99.9% confidence is miscalibrated. Calibration focuses on the quality of the uncertainty estimate, not the correctness of the top prediction. In risk-critical applications like medical diagnosis or autonomous driving, a calibrated low-confidence prediction is often more valuable than an overconfident wrong answer, as it can trigger a human-in-the-loop intervention.

The primary metric for measuring logit calibration quality. ECE partitions predictions into M equal-width bins based on confidence, then computes the weighted average of the absolute difference between accuracy and confidence in each bin. A perfectly calibrated model has an ECE of zero. For example, if a bin contains 100 predictions with 0.8 confidence, exactly 80 should be correct. ECE is the standard benchmark for comparing calibration methods like temperature scaling and Platt scaling.

The most widely adopted logit calibration technique due to its simplicity and effectiveness. A single scalar parameter T is learned on a validation set and applied to divide all logits before the softmax function:

• T > 1: Softens probabilities, reducing overconfidence
• T < 1: Sharpens probabilities, increasing confidence separation
• T = 1: Original probabilities unchanged Unlike Platt scaling or isotonic regression, temperature scaling preserves the rank order of predictions and does not alter accuracy—only confidence scores.

Logit calibration primarily addresses epistemic uncertainty—the reducible uncertainty from limited training data or model capacity. This is distinct from aleatoric uncertainty, which is irreducible noise inherent in the data itself (e.g., ambiguous images, overlapping classes). A well-calibrated model should:

• Express low confidence on out-of-distribution inputs (high epistemic uncertainty)
• Express appropriate confidence on noisy but in-distribution inputs (high aleatoric uncertainty) Conflating these two uncertainty types leads to miscalibration and unreliable confidence estimates.

A distribution-free alternative to logit calibration that provides rigorous mathematical guarantees. Instead of adjusting confidence scores, conformal prediction generates prediction sets with a guaranteed coverage probability (e.g., 90% of true labels fall within the set). Key advantages:

• Model-agnostic: Works with any classifier
• Finite-sample validity: Guarantees hold without asymptotic assumptions
• No retraining required: Uses a held-out calibration set Conformal prediction is gaining traction in high-stakes domains where calibrated probabilities alone are insufficient for decision-making.

The degradation of logit calibration quality over time as data distributions shift in production. A model calibrated on training data may become severely miscalibrated when encountering concept drift or covariate shift. Monitoring strategies include:

• Tracking ECE on live prediction batches
• Detecting when confidence distributions diverge from baseline
• Implementing online recalibration with sliding windows Calibration drift is a critical MLOps concern, as undetected miscalibration leads to overconfident errors in autonomous systems.

A parametric calibration method that fits a logistic regression model on top of a classifier's raw scores. Unlike temperature scaling, Platt scaling learns two parameters (weight and bias) and can change the rank order of predictions. It was originally developed for Support Vector Machines but generalizes to neural networks. Platt scaling works best when calibration errors are symmetric across the probability range, but can underperform on modern deep networks where temperature scaling is preferred for its simplicity and rank-preserving property.