Temperature Scaling

Temperature scaling introduces exactly one scalar parameter (T) that divides all logits before the softmax function. This parameter is optimized on a held-out validation set to minimize the Negative Log-Likelihood (NLL) .

• T > 1: Softens probabilities, reducing overconfidence
• T < 1: Sharpens probabilities, increasing confidence separation
• T = 1: Recovers the original uncalibrated model

The simplicity of a single parameter prevents overfitting to the calibration set, unlike multi-parameter methods such as Platt scaling or isotonic regression.

A critical property of temperature scaling is that it is accuracy-preserving. Because the temperature parameter is a monotonic transformation applied uniformly across all logits, the relative ordering of class predictions never changes.

• The argmax (top-1 prediction) remains identical before and after scaling
• Only the confidence score associated with that prediction is adjusted
• This makes it safe to apply post-deployment without altering model behavior

This contrasts with label smoothing, which changes the training objective and can alter the decision boundary.

While originally demonstrated for binary and multiclass classification, temperature scaling extends naturally to more granular calibration criteria.

• Classwise Calibration: A single temperature parameter may not achieve per-class calibration. A matrix scaling variant learns a temperature vector (one T per class) for finer control
• Top-Label Calibration: Standard single-parameter temperature scaling is often sufficient to achieve well-calibrated maximum probabilities
• Modern implementations in frameworks like PyTorch and TensorFlow provide built-in support for temperature scaling as a post-processing layer

Temperature scaling shares a mathematical foundation with knowledge distillation. In distillation, a high temperature is used to soften the teacher model's output distribution, revealing dark knowledge about inter-class similarities.

• The softened distribution provides richer training signals than hard labels
• The same temperature parameter controls the entropy of the softmax output
• In calibration, the goal is to match confidence to empirical accuracy; in distillation, the goal is to transfer representational knowledge

This connection highlights temperature as a fundamental mechanism for controlling predictive uncertainty in deep learning.

Despite its effectiveness, temperature scaling has known limitations that practitioners must consider.

• Underconfidence: On shifted or out-of-distribution data, a temperature optimized on in-distribution validation may produce underconfident predictions
• Non-Identifiability: A single scalar cannot correct complex miscalibration patterns, such as when a model is overconfident in one region and underconfident in another
• Dependence on Validation Set: The optimal T is sensitive to the size and representativeness of the calibration set

For these scenarios, deep ensembles or conformal prediction may provide more robust uncertainty estimates.

Implementing temperature scaling requires a disciplined validation protocol to avoid data leakage.

• Data Split: Use a three-way split: training set for model weights, validation set for learning T, and test set for evaluating calibration with Expected Calibration Error (ECE)
• Optimization: T is typically initialized to 1 and optimized using gradient descent or L-BFGS on the NLL of the validation set
• Framework: In PyTorch, this is a simple nn.Parameter multiplied with logits before F.softmax

A reliability diagram should always be plotted before and after scaling to visually confirm calibration improvement.