Temperature Scaling

In clinical AI, a model's confidence score must reflect true diagnostic accuracy. A model predicting '85% probability of malignancy' should be correct 85% of the time. Temperature scaling adjusts overconfident or underconfident logits post-training, ensuring radiologists receive reliable risk assessments. This is critical for triage systems and decision support tools where confidence directly impacts patient pathways.

Robotic and autonomous vehicle systems use classification models for perception (e.g., object identification). A well-calibrated model allows the system to accurately gauge its own uncertainty. For instance, if a model is only 60% confident it sees a pedestrian, the system can trigger a more cautious control policy or request human intervention. Temperature scaling provides a computationally cheap way to achieve this calibrated uncertainty without retraining the core vision model.

In algorithmic trading or credit scoring, predicted probabilities drive monetary decisions. A miscalibrated model can systematically overestimate or underestimate risk. By applying temperature scaling on a held-out calibration set, firms can ensure a predicted '1% probability of default' is empirically true. This leads to more accurate expected value calculations, better portfolio allocation, and compliance with model risk management (MRM) regulations that demand reliable probability estimates.

LLMs often generate overconfident but incorrect statements. For tasks like multiple-choice QA or factual claim generation, temperature scaling can be applied to the logits of the final token prediction to better calibrate the model's confidence in its answer. This improves the utility of confidence thresholds for filtering outputs, making RAG systems and agentic workflows more reliable by allowing them to reject low-confidence, potentially hallucinated responses.

In a model cascade, a fast, cheap model filters easy cases, passing only hard ones to a more accurate but expensive model. This requires the first model's confidence scores to be perfectly calibrated to avoid passing too many or too few cases. Temperature scaling optimizes this threshold. Similarly, for model ensembles, averaging miscalibrated probabilities remains miscalibrated. Scaling each model's logits with its optimal temperature before softmax and averaging yields a better-calibrated ensemble prediction.

Temperature scaling is a staple in production calibration pipelines due to its simplicity (one parameter) and stability. It is applied as a final step before model deployment and is periodically re-fitted on fresh calibration data to combat calibration drift. Its low overhead makes it ideal for continuous deployment environments where models are frequently updated, as it requires only a forward pass on a calibration set to recompute the optimal temperature, unlike more complex methods like isotonic regression.

Platt scaling (or sigmoid calibration) is a parametric, post-hoc calibration method for binary classifiers. It fits a logistic regression model with two parameters (a weight and a bias) to the classifier's logits, transforming them into calibrated probabilities via the sigmoid function. It is more flexible than temperature scaling but requires a separate calibration dataset and risks overfitting if not regularized.

• Key Use Case: Calibrating outputs from models like SVMs or boosted trees that don't natively output probabilities.
• Comparison to Temperature Scaling: Uses two parameters instead of one, offering more flexibility but less stability on small calibration sets.

Isotonic regression is a non-parametric, post-hoc calibration method that fits a piecewise constant, non-decreasing function to map a classifier's raw outputs to calibrated probabilities. It makes minimal assumptions about the underlying score distribution and can model complex, non-linear miscalibration patterns.

• Key Use Case: Correcting severe, non-linear miscalibration where simpler parametric methods fail.
• Trade-off: High flexibility comes at the cost of requiring larger calibration datasets to avoid overfitting and increased computational complexity compared to temperature scaling.

Expected Calibration Error (ECE) is a primary scalar metric for quantifying miscalibration. It works by:

1. Binning predictions based on their confidence score (e.g., 0-0.1, 0.1-0.2).
2. For each bin, calculating the difference between the average confidence (predicted accuracy) and the actual empirical accuracy.
3. Taking a weighted average of these absolute differences.

A lower ECE indicates better calibration. It is the standard benchmark for evaluating techniques like temperature scaling.

A reliability diagram is the fundamental visual diagnostic tool for calibration. It plots a model's average predicted confidence (on the x-axis) against its observed empirical accuracy (on the y-axis) across multiple confidence bins.

• Perfect Calibration: Points fall on the diagonal line (accuracy = confidence).
• Visualizing Miscalibration: Points below the diagonal indicate overconfidence; points above indicate underconfidence.
• Primary Use: Provides an intuitive, graphical assessment that complements scalar metrics like ECE, showing how a model is miscalibrated.

The Brier score is a proper scoring rule that measures the mean squared error between a model's predicted probabilities and the true binary outcomes (e.g., 0 or 1). It decomposes into two components: calibration loss and refinement loss (also called sharpness).

• Formula: (BS = \frac{1}{N}\sum_{t=1}^{N}(f_t - o_t)^2)
• Interpretation: A lower Brier score is better. It evaluates both how well-calibrated the probabilities are (calibration) and how decisive they are (refinement). Unlike ECE, it is sensitive to the entire probability vector, not just the confidence of the predicted class.

Post-hoc calibration is the overarching category of techniques applied to a trained model's outputs to improve probability calibration, without modifying the model's internal parameters. Temperature scaling, Platt scaling, and isotonic regression are all post-hoc methods.

• Core Principle: Decouple the task of achieving high accuracy from the task of achieving calibrated uncertainties.
• Workflow:
  1. Train a model for maximum accuracy.
  2. Reserve a separate calibration set.
  3. Use the calibration set to learn a simple function that maps the model's raw outputs to better-calibrated probabilities.
• Advantage: Simple, computationally cheap, and highly effective for modern neural networks.