Temperature Scaling

Temperature scaling introduces exactly one learnable parameter (T > 0) that is optimized on a held-out validation set using negative log-likelihood loss. Unlike more complex calibration methods such as Platt scaling or isotonic regression, this simplicity prevents overfitting to the calibration set while effectively correcting systematic overconfidence in modern deep neural networks.

• The parameter T is applied uniformly across all classes
• T > 1 softens the distribution, reducing peak confidence
• T < 1 sharpens the distribution, increasing peak confidence
• T = 1 leaves the original probabilities unchanged
• Optimization uses the same NLL loss as the original training

A critical property of temperature scaling is that it is rank-preserving. Because the temperature parameter T is a strictly positive scalar divisor applied uniformly to all logits, the relative ordering of class probabilities remains unchanged. The model's Top-1 accuracy and Top-K accuracy are mathematically guaranteed to remain identical before and after scaling.

• The argmax operation is invariant to positive scaling
• Decision boundaries are unaffected
• Only the confidence values associated with predictions change
• This makes it safe to apply post-deployment without regression testing

Temperature scaling directly minimizes the Expected Calibration Error (ECE), the primary metric for measuring calibration quality. ECE quantifies the gap between predicted confidence and empirical accuracy by partitioning predictions into bins and computing the weighted average of the absolute difference between accuracy and confidence within each bin.

• ECE = Σ (bin_size / N) × |accuracy(bin) - confidence(bin)|
• Modern neural networks often exhibit ECE values of 5-15%
• Temperature scaling typically reduces ECE to below 2%
• The improvement is most pronounced in high-confidence bins
• Works exceptionally well with ReLU-based architectures prone to overconfidence

Temperature scaling is applied after training is complete and requires no modification to the model architecture, training procedure, or loss function. This non-intrusive nature makes it ideal for calibrating pre-trained models in production environments where retraining is impractical or prohibitively expensive.

• No access to training data required—only a small validation set
• Compatible with any model that outputs logits
• Can be applied to frozen checkpoints and third-party models
• Integrates seamlessly into inference pipelines
• Commonly used in RAG systems to calibrate retrieval confidence scores

Temperature scaling shares a mathematical foundation with knowledge distillation, where a high-temperature softmax is used to extract richer supervisory signals from a teacher model. In both cases, elevated temperatures reveal the dark knowledge encoded in the relative probabilities of incorrect classes, exposing inter-class similarities that are collapsed at T=1.

• Distillation uses T >> 1 during training to soften targets
• Calibration uses T optimized for ECE minimization at inference
• Both leverage the same logit-scaling mechanism
• The optimal calibration temperature is typically between 1.5 and 3.0
• This connection explains why distilled models often exhibit better calibration

Temperature scaling assumes that miscalibration is a global, systematic bias that can be corrected with a single scalar. It fails when different regions of the input space require different temperature values, such as when a model is overconfident on in-distribution data but underconfident on out-of-distribution samples.

• Cannot correct per-class miscalibration patterns
• Ineffective when the model exhibits both over- and under-confidence
• Does not address dataset shift or covariate shift
• Matrix scaling (class-wise temperatures) can help but risks overfitting
• For severe miscalibration, consider isotonic regression or Platt scaling

A parametric calibration method that fits a logistic regression model on top of raw classifier scores. Unlike temperature scaling, Platt scaling applies an affine transformation (a * logit + b) rather than a single scalar divisor. It is effective for binary classification but can distort multiclass distributions. Originally developed for Support Vector Machines, it remains a baseline for confidence calibration in legacy systems.

A non-parametric calibration method that learns a piecewise constant, monotonically increasing function mapping raw scores to calibrated probabilities. It offers greater flexibility than temperature scaling but is prone to overfitting on small validation sets. Useful when the relationship between logits and empirical accuracy is non-linear. Often compared against temperature scaling in reliability diagram benchmarks.

The primary metric for evaluating probabilistic calibration. ECE partitions predictions into M equally-spaced bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin. A perfectly calibrated model has an ECE of zero. Temperature scaling directly minimizes the Negative Log-Likelihood (NLL) on a held-out validation set, which indirectly reduces ECE.

The core mechanism modified by temperature scaling. The standard softmax function converts logits z_i into probabilities using exp(z_i) / Σ exp(z_j). Temperature scaling introduces a scalar T > 0: exp(z_i / T) / Σ exp(z_j / T). When T > 1, the distribution softens (higher entropy). When T < 1, it sharpens (peaked distribution). This is distinct from the temperature hyperparameter used in text generation sampling.

A downstream application of calibrated probabilities in re-ranking pipelines. After temperature scaling produces reliable confidence scores, a threshold is set to filter out low-confidence predictions. Documents with calibrated scores below the threshold are discarded before final answer generation. This prevents the generator from hallucinating on uncertain retrievals. Thresholds are tuned against precision-recall curves on validation data.

The process of transforming raw relevance scores from heterogeneous sources onto a common scale before fusion. In a two-stage retrieval pipeline, temperature scaling can serve as a normalization step, converting cross-encoder logits into well-calibrated probabilities. This allows meaningful comparison and aggregation with scores from other rankers using techniques like CombSUM or Reciprocal Rank Fusion (RRF).