Temperature Scaling

Temperature scaling operates by dividing the logits (the raw, unnormalized outputs from a model's final layer) by a learned scalar parameter T (temperature) before applying the softmax function.

• Formula: softmax(z_i / T) where z_i are the logits.
• Effect: When T > 1, the output probability distribution becomes 'softer' (more uniform), reducing overconfidence. When T < 1, the distribution becomes 'sharper' (more peaky), increasing confidence.
• The optimal temperature T* is found by minimizing negative log-likelihood (NLL) on a separate validation set, distinct from the training set.

Its primary advantage is extreme simplicity. Unlike other calibration methods, it introduces only one global parameter to tune.

• Efficiency: The optimization for T is a convex problem on the validation set, typically solved quickly with a method like gradient descent.
• Preservation of Accuracy: Because it applies a monotonic transformation to the logits, it does not change the model's predicted class ranking. The argmax of the probabilities remains unchanged, meaning classification accuracy is preserved while calibration is improved.
• This makes it a highly efficient, low-risk first step in any calibration pipeline.

The core goal is to correct miscalibration, where a model's predicted confidence does not match its true likelihood of being correct (e.g., predicting class A with 90% confidence but being right only 70% of the time).

• Corrects Overconfidence: Modern neural networks, especially large ones, are frequently overconfident. A temperature T > 1 (common) systematically scales down high confidences.
• Measured by ECE: Improvement is quantified by metrics like Expected Calibration Error (ECE), which bins predictions by confidence and measures the gap between average confidence and accuracy within each bin. Temperature scaling directly minimizes this gap.
• It primarily addresses confidence sharpness, not underlying model uncertainty.

While powerful for its simplicity, temperature scaling has defined boundaries.

• Cannot Fix All Miscalibration: It assumes the model's miscalibration is isotropic (similar across all confidence levels and classes). It cannot correct more complex, class-specific miscalibration patterns that methods like Platt Scaling (which uses a logistic regression per class) might address.
• Does Not Improve Accuracy: It is a post-hoc method. It cannot improve the model's fundamental discriminative power or correct systematic errors in its predictions.
• Separate from Uncertainty Quantification: It adjusts the scale of existing probabilities but does not generate new measures of epistemic uncertainty (model uncertainty) like Bayesian Neural Networks or Deep Ensembles do.

Temperature scaling is a foundational block within a broader calibration and uncertainty toolkit.

• Vs. Platt Scaling: Platt scaling fits a logistic regression to logits, offering more flexibility (slope & intercept) but risks overfitting on small validation sets. Temperature scaling is more constrained and stable.
• Complement to Ensembles: It is often applied to each member of a Deep Ensemble before averaging, calibrating the individual models first.
• Precursor to Selective Classification: A well-calibrated confidence score from temperature scaling is crucial for selective classification (rejection option), where the model abstains on low-confidence inputs.
• Used with Label Smoothing: Models trained with label smoothing (a regularization technique) often already produce better-calibrated logits, which temperature scaling can further refine.

Implementing temperature scaling involves a straightforward, three-step process.

1. Train Model: Train your classifier as usual on the training set.
2. Learn Temperature: On a held-out validation set (not used for training):
   • Collect the model's logits and the true labels.
   • Optimize the scalar temperature parameter T by minimizing the Negative Log-Likelihood (NLL) loss. This is typically a one-dimensional optimization problem.
3. Apply at Inference: During deployment, divide all output logits by the learned T before applying the softmax function to obtain calibrated probabilities.

• Critical Note: The test set must be completely unseen during both training and temperature learning to obtain an unbiased evaluation of calibration performance.

Calibration error measures the discrepancy between a model's predicted confidence scores and its actual empirical accuracy. A perfectly calibrated model's confidence of 0.8 should correspond to an 80% chance of being correct. Expected Calibration Error (ECE) is the most common scalar metric, calculated by binning predictions by confidence and averaging the absolute difference between average confidence and accuracy within each bin. High calibration error indicates overconfidence or underconfidence, which temperature scaling aims to correct.

Platt scaling is a post-hoc calibration method that fits a logistic regression model to a classifier's raw scores (logits) on a held-out validation set to produce better-calibrated probability estimates. Unlike temperature scaling, which uses a single scalar parameter, Platt scaling learns two parameters (a slope and an intercept). It is more flexible but can overfit on small calibration sets. It is most effective for binary classification and is the precursor to modern multi-class extensions like temperature scaling.

Selective classification, or classification with a rejection option, is a paradigm where a model is allowed to abstain from making a prediction when its confidence is below a chosen threshold. This is critical for high-stakes applications. Temperature scaling directly improves the reliability of the confidence scores used for this abstention decision. The trade-off is visualized via a risk-coverage curve, which plots error rate against the fraction of samples the model chooses to predict on.

Expected Calibration Error (ECE) is the primary metric for quantifying miscalibration. It is calculated by:

• Partitioning predictions into M bins based on their predicted confidence.
• For each bin, calculating the absolute difference between the average confidence and the average accuracy.
• Taking a weighted average of these differences, weighted by the number of samples in each bin. A lower ECE indicates better calibration. Temperature scaling is explicitly optimized to minimize ECE on a validation set.

Uncertainty Quantification (UQ) is the broader field of measuring and interpreting the uncertainty in model predictions. It distinguishes between:

• Aleatoric uncertainty: Irreducible noise inherent in the data.
• Epistemic uncertainty: Reducible uncertainty from a lack of model knowledge. Temperature scaling is a calibration technique that refines a model's confidence estimates but does not distinguish between these types. Methods like Bayesian Neural Networks (BNNs), Monte Carlo Dropout, and Deep Ensembles provide more comprehensive UQ.

Negative Log-Likelihood (NLL), or log loss, is a proper scoring rule used as a training objective. It penalizes a model based on the negative logarithm of the probability it assigns to the true label. Minimizing NLL encourages both accuracy and calibration. While models are often trained with NLL, they frequently become miscalibrated. Temperature scaling is applied post-training to a model's logits to further minimize NLL on a calibration set, improving the probabilistic quality of its outputs without retraining.