Temperature Scaling

Temperature scaling modifies the standard softmax function by dividing logits by a scalar T before exponentiation. When T = 1, the function behaves as standard softmax. As T → ∞, the distribution approaches uniform, making all classes equally probable. As T → 0, the distribution becomes a Dirac delta at the argmax, outputting hard labels. The formula is: softmax(z_i / T) = exp(z_i / T) / Σ_j exp(z_j / T). This single scalar parameter is applied identically across all classes, preserving the rank order of the original logits while adjusting the confidence of the prediction.

The term dark knowledge refers to the information encoded in the relative probabilities of incorrect classes within a teacher model's softmax output. At high temperatures, the softmax surface reveals inter-class similarities that are obscured at T=1. For example, a model classifying a handwritten '3' might assign a probability of 10⁻⁶ to '8' and 10⁻⁹ to 'dog' at T=1. With temperature scaling, these differences are magnified, showing that '3' is semantically closer to '8' than to 'dog'. This structural knowledge about the data manifold is transferred to the student during distillation.

During knowledge distillation, the student model is trained with a composite loss function that balances two objectives. The distillation loss uses the Kullback-Leibler divergence between the student's soft targets and the teacher's soft targets, both computed with the same high temperature T. The student loss is the standard cross-entropy between the student's predictions at T=1 and the ground-truth hard labels. A weighting parameter α (typically 0.1 to 0.5) controls the relative contribution of these two losses: L_total = α * L_hard + (1 - α) * T² * L_soft. The T² scaling factor is applied to keep gradient magnitudes consistent.

Beyond distillation, temperature scaling serves as a post-hoc calibration method for modern neural networks. Unlike Platt scaling or isotonic regression, temperature scaling does not change the model's accuracy or the rank ordering of predictions—it only adjusts confidence estimates. A single scalar T is optimized on a held-out validation set to minimize the negative log-likelihood or expected calibration error (ECE). This technique is particularly effective for calibrating large language models and deep ensembles, where overconfident predictions are a well-documented problem. The optimal T is found via gradient descent on the calibration metric.

Temperature scaling directly controls the entropy of the softmax distribution, which in turn affects the gradient flow during backpropagation. At low temperatures, gradients are sparse—only the top class receives a meaningful signal. At high temperatures, the gradient is distributed more evenly across classes, providing richer training signals for the student. However, excessively high temperatures can drown the signal in noise. Empirical studies show optimal distillation temperatures typically range from T=2 to T=20, depending on the number of classes and the complexity of the teacher model. The choice of T represents a bias-variance tradeoff in the transferred knowledge.

An important theoretical property of temperature scaling is its relationship to logit matching. When the distillation loss uses KL divergence with a high temperature T and the soft targets are derived from the teacher's logits, the gradient of the distillation loss with respect to the student's logits approaches a simple squared error between the student and teacher logits as T → ∞. Specifically, the gradient becomes proportional to (z_student - z_teacher) / T. This reveals that high-temperature distillation is asymptotically equivalent to directly matching the logit vectors of the teacher, providing a simpler interpretation of what the student is learning.