Temperature Scaling

Temperature scaling introduces exactly one learnable parameter (T > 0) applied to all logits before the softmax function. Unlike more complex methods such as Platt scaling or isotonic regression, it does not modify the model's internal architecture or require auxiliary models. This makes it computationally trivial to implement and resistant to overfitting on small calibration sets. The parameter T is optimized on a held-out validation set by minimizing the negative log-likelihood (NLL) with respect to the ground-truth labels.

A critical property of temperature scaling is that it is accuracy-preserving. Multiplying all logits by a scalar does not change their relative ordering, meaning the model's top-1 prediction and overall classification accuracy remain completely unchanged. This is essential for safety-critical applications where you cannot afford to alter the model's decisions, only the confidence associated with them. The method strictly adjusts the sharpness of the distribution, not the decision boundary.

The temperature parameter T directly controls the entropy of the output distribution:

• T > 1 (Softening): Logits are divided by a large number, making the probability distribution more uniform. This corrects overconfident models that output probabilities like 0.99 for incorrect predictions.
• T < 1 (Sharpening): Logits are divided by a small number, making the distribution peakier. This corrects underconfident models.
• T = 1: The original probability distribution is recovered.

Temperature scaling directly minimizes the Expected Calibration Error (ECE), the primary metric for measuring miscalibration. ECE partitions predictions into confidence bins and computes the weighted absolute difference between average confidence and accuracy. By smoothing overconfident peaks, temperature scaling ensures that when a model says it is 80% confident, it is actually correct roughly 80% of the time. Modern neural networks, particularly deep ensembles, often see ECE reduced by an order of magnitude after scaling.

It is crucial to distinguish calibration temperature from the temperature hyperparameter used in knowledge distillation. In distillation, a high temperature is used during training to extract 'dark knowledge' from a teacher model's soft labels. In calibration, the temperature is a post-hoc correction factor learned after training is complete. While both use the same mathematical operation, their purpose and placement in the workflow are entirely distinct.

Standard temperature scaling assumes miscalibration is uniform across all classes and input regions, which is often not true. Extensions like matrix scaling (class-specific temperatures) and vector scaling (per-class plus a bias) offer more flexibility but risk overfitting. For modern LLMs, context-aware calibration is an active research area, as a single global T is insufficient to correct the complex confidence errors arising from in-context learning and long-form generation.

The primary empirical metric for evaluating how well a model's confidence scores align with its actual accuracy. ECE partitions predictions into M equally-spaced bins based on confidence and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• A perfectly calibrated model has an ECE of 0
• Modern neural networks often exhibit overconfidence, producing high ECE scores
• Used as the standard benchmark to measure the effectiveness of Temperature Scaling and other calibration methods

A family of post-processing techniques that adjust the raw, unnormalized output scores (logits) of a neural network before the softmax function is applied. Unlike training-time interventions, logit calibration operates on a held-out validation set without modifying model weights.

• Temperature Scaling is the simplest and most widely adopted member of this family
• Vector Scaling extends the concept with class-specific parameters
• Matrix Scaling applies a full linear transformation to the logit vector
• All methods optimize parameters to minimize a proper scoring rule like Negative Log-Likelihood (NLL) on a calibration set

A quantitative probability assigned by a model to indicate the likelihood that its prediction is correct. In classification, this is typically the maximum softmax probability after Temperature Scaling is applied.

• Raw confidence scores from modern deep networks are often miscalibrated—a 0.99 confidence may only correspond to 0.80 accuracy
• Temperature Scaling directly transforms these scores to better reflect true correctness likelihood
• Well-calibrated confidence scores are critical for downstream decision-making in high-stakes applications like medical diagnosis and autonomous driving

Two fundamental categories of uncertainty that Temperature Scaling helps disentangle:

• Epistemic Uncertainty: The reducible uncertainty caused by limited knowledge or training data. Can be decreased with more data. Temperature Scaling partially addresses this by correcting model overconfidence on out-of-distribution inputs
• Aleatoric Uncertainty: The irreducible statistical noise inherent in the data itself, such as label noise or overlapping class distributions
• Temperature Scaling primarily corrects for epistemic miscalibration but cannot eliminate aleatoric uncertainty
• Understanding this distinction is essential for building trustworthy AI systems that know when they don't know

A model-agnostic framework that provides mathematically rigorous prediction sets with guaranteed coverage probability, offering an alternative to single-point confidence scores.

• Produces a set of possible labels rather than one prediction, e.g., "{cat, dog}" with 95% guaranteed coverage
• Unlike Temperature Scaling, conformal prediction makes no assumptions about the underlying model's calibration
• Can be combined with Temperature Scaling: first calibrate the model, then apply conformal prediction for formal guarantees
• Increasingly used in regulated industries where provable error bounds are required

The degradation over time of a model's calibration quality as the data distribution shifts from the training environment. A model perfectly calibrated at deployment may become severely miscalibrated in production.

• Caused by covariate shift, label shift, or concept drift in the input stream
• Temperature Scaling parameters learned on a static validation set may become suboptimal
• Requires continuous monitoring of ECE in production and periodic recalibration
• A critical concern for long-running enterprise AI systems where trustworthiness must be maintained over months or years