Temperature Scaling

Temperature scaling introduces exactly one parameter (T > 0) that divides all logits before the softmax function. This is the simplest form of Platt scaling for multi-class problems.

• T = 1: The original, uncalibrated model output
• T > 1: Softens probabilities, reducing overconfidence
• T < 1: Sharpens probabilities, increasing confidence peaks

The parameter is learned by minimizing the Negative Log-Likelihood (NLL) on a held-out validation set, completely separate from the training data.

A critical property of temperature scaling is that it is a monotonic transformation of the logits. Because the temperature parameter divides all logits equally, the relative ordering of class probabilities never changes.

• The argmax prediction remains identical before and after scaling
• Top-k accuracy is completely unaffected
• Only the confidence values associated with predictions are adjusted

This makes it safe to apply to production models without any risk of degrading classification performance.

Modern deep neural networks are notoriously miscalibrated, often producing confidence scores that are systematically higher than their actual accuracy. Temperature scaling directly addresses this.

• A model might predict a class with 95% confidence but only be correct 80% of the time
• By learning T > 1, the probability mass is spread more evenly across classes
• The resulting probabilities better reflect the empirical frequency of correctness

This is essential for high-stakes applications where a calibrated "I don't know" is more valuable than an overconfident error.

The optimal temperature T is found by optimizing a proper scoring rule on a dedicated calibration set. This set must be distinct from both the training data and the final test data to avoid biased estimates.

• Objective: Minimize NLL on the calibration set
• Optimizer: Typically L-BFGS or simple grid search since there is only one parameter
• Data Size: A few thousand samples are usually sufficient

The learned T generalizes well because it captures the global tendency of the model to be overconfident or underconfident.

Temperature scaling occupies a sweet spot between complexity and effectiveness in the calibration landscape.

• vs. Histogram Binning: Temperature scaling is parametric and smoother, avoiding discontinuous probability jumps
• vs. Isotonic Regression: Much less prone to overfitting on small calibration sets
• vs. Bayesian Methods: Computationally trivial, requiring no multiple forward passes or weight distributions
• vs. Ensemble Methods: Does not require training multiple models

For most practical applications, temperature scaling is the recommended baseline before exploring more complex approaches.

Despite its elegance, temperature scaling has important limitations that practitioners must understand.

• Global parameter: A single T cannot correct class-specific or input-dependent miscalibration patterns
• No effect on ranking: Cannot fix cases where the model confidently predicts the wrong class
• Underconfidence: If the model is already underconfident, T < 1 can amplify noise
• Out-of-distribution: Does not help detect inputs far from the training distribution; the model may still be confidently wrong

For these scenarios, consider matrix scaling (per-class temperatures) or vector scaling as extensions.

A scalar summary statistic that measures the discrepancy between a model's confidence and its observed accuracy. ECE works by partitioning predictions into M equally-spaced bins and computing the weighted average of the absolute difference between accuracy and confidence within each bin:

• Perfect calibration: ECE = 0
• Overconfidence: Confidence exceeds accuracy (positive gap)
• Underconfidence: Accuracy exceeds confidence (negative gap)

Temperature scaling directly minimizes the ECE metric on a validation set.

Model uncertainty arising from a lack of knowledge or data, which can theoretically be reduced with more training samples or a better model architecture. Unlike aleatoric uncertainty, epistemic uncertainty is highest in regions of sparse training data. Temperature scaling primarily addresses confidence calibration rather than decomposing uncertainty types, but it complements methods like Deep Ensembles and Monte Carlo Dropout that explicitly model epistemic uncertainty.

Inherent and irreducible noise in the data generation process, such as measurement error or inherent stochasticity. This type of uncertainty cannot be reduced by collecting more data. Examples include:

• Sensor noise in IoT deployments
• Label ambiguity in medical imaging
• Inherent randomness in financial markets

Temperature scaling does not distinguish between aleatoric and epistemic uncertainty, which is why advanced deployments often combine it with Bayesian Neural Networks.

A strictly proper scoring rule that measures the mean squared difference between the predicted probability and the actual binary outcome. Unlike ECE, the Brier Score is a proper scoring rule, meaning its optimal value is achieved only by the true conditional probability. Key properties:

• Range: 0 (perfect) to 1 (worst)
• Decomposes into refinement and calibration components
• Penalizes both overconfidence and underconfidence

Temperature scaling optimizes the log-likelihood, which is also a proper scoring rule.

A method for quantifying predictive uncertainty by training multiple neural networks with different random initializations and averaging their outputs. Deep ensembles capture epistemic uncertainty through model disagreement. When combined with temperature scaling:

• Ensembles provide diverse predictions reflecting model uncertainty
• Temperature scaling calibrates the aggregated output
• The combination achieves state-of-the-art calibration on benchmarks like CIFAR-100 and ImageNet