Temperature Scaling

The temperature (T) is a single, positive scalar hyperparameter applied to the logits before the softmax function. It directly controls the 'softness' of the output probability distribution.

• T = 1: The standard softmax function. No change to the original model output.
• T > 1: The distribution is softened. Probabilities become more uniform, revealing the model's relative confidence across classes (the 'dark knowledge').
• T < 1: The distribution is sharpened, making the model more confident (peaky). This can exaggerate overconfidence and is rarely used in distillation.

The modified softmax is: softmax(z_i / T), where z are the logits.

In knowledge distillation, a high temperature (e.g., T=3 to T=10) is applied to the teacher model's logits to create soft targets. These softened probabilities provide a richer training signal than hard one-hot labels.

• The student model is trained to match these soft targets using a loss function like Kullback-Leibler (KL) Divergence.
• The softened probabilities encode inter-class similarities. For example, an image of a '7' might have a non-zero probability for '1', teaching the student about shared features.
• During final inference, the student uses a temperature of T=1, recovering a standard, sharp probability distribution.

Separate from distillation, temperature scaling is a highly effective post-hoc calibration method. A poorly calibrated model's predicted confidence does not match its true accuracy (e.g., predicting 0.9 confidence but being right only 70% of the time).

• After training, a validation set is used to optimize the temperature parameter T to minimize the gap between confidence and accuracy.
• It is a monotonic transformation that does not change the model's predicted class ranking (argmax), only the confidence scores.
• It is computationally cheap, requiring optimization of just one parameter, and often outperforms more complex calibration methods.

The core operation is a simple scaling of the logit vector z before applying the softmax function.

Softmax with Temperature: σ(z, T)_i = exp(z_i / T) / Σ_j exp(z_j / T)

Key Properties:

• As T → ∞, the output distribution approaches uniform (1/K for K classes).
• As T → 0, the output approaches a one-hot distribution at the highest logit.
• The gradients passed to the student during distillation are scaled by 1/T^2, providing a more stable and gentle learning signal when T > 1.

Implementing temperature scaling is straightforward in modern deep learning frameworks.

For Distillation:

1. Forward pass teacher batch with high T to get soft targets.
2. Forward pass student batch with the same high T.
3. Compute KL divergence loss between the two softened distributions.
4. Combine with a standard cross-entropy loss using hard labels (often with a weighting factor α).

For Calibration:

1. Train model as normal.
2. On a held-out validation set, optimize T to minimize Negative Log Likelihood (NLL) or Expected Calibration Error (ECE).
3. Apply the optimized T to scale logits during all future inference.

While powerful, temperature scaling has specific constraints.

• Single-Parameter Simplicity: Its strength is also a limitation. It cannot correct calibration errors that require non-monotonic adjustments across different parts of the input space.
• Dataset Dependency: The optimal T for distillation or calibration is dataset and model-dependent and must be validated.
• Logits Requirement: It requires access to the model's logits, not just the final class prediction. Some APIs may not expose these.
• No Architectural Change: It does not modify the model's weights or architecture; it is a inference-time adjustment. For complex miscalibration, methods like Platt scaling or isotonic regression may be more flexible.

Temperature scaling's primary application is in the teacher-student framework. By applying a temperature T > 1 to the teacher's softmax, the output distribution is softened, revealing dark knowledge—the relative similarities between incorrect classes. This richer signal, compared to hard one-hot labels, provides a more informative training target for the student model. For example, in image classification, a softened output for a 'cat' image might show high probability for 'cat', moderate for 'lynx', and low for 'truck', teaching the student about semantic relationships within the data.

A critical secondary use is post-hoc calibration of neural networks. Modern models, especially large ones, are often miscalibrated—their predicted confidence scores do not match their true likelihood of being correct (e.g., predicting 0.9 confidence but being right only 70% of the time). Temperature scaling is applied as a single-parameter transformation on the logits of a trained, frozen model on a validation set. An optimal T is found to minimize the gap between confidence and accuracy, improving metrics like Expected Calibration Error (ECE). This is essential for high-stakes applications like medical diagnosis or autonomous driving, where confidence scores must be trustworthy.

In large language model inference, temperature scaling directly controls the randomness or creativity of generated text. It is applied to the logits before sampling the next token.

• Low Temperature (T < 1): Sharpens the distribution, making high-probability tokens more likely. Output becomes more deterministic, focused, and repetitive.
• High Temperature (T > 1): Flattens the distribution, giving lower-probability tokens a better chance. Output becomes more diverse, creative, and potentially nonsensical. For example, code generation typically uses a low T (~0.2) for deterministic, correct syntax, while creative writing might use a higher T (~0.8-1.2). This parameter is a key lever in prompt engineering for steering model behavior.

Temperature scaling is a foundational component enabling more sophisticated distillation techniques:

• Online Distillation: Used in frameworks like Deep Mutual Learning, where peer models teach each other using softened targets generated in real-time during co-training.
• Self-Distillation: In architectures like Born-Again Networks, a model distills knowledge from its own earlier training iterations, relying on temperature-scaled outputs from its previous checkpoint as soft labels.
• Data-Free Distillation: When the original training data is unavailable, temperature-scaled outputs from the teacher can help generate or guide the creation of synthetic data for student training.
• Quantization-Aware Distillation (QAT): Temperature-scaled soft targets can be used while simulating quantization noise, helping the student learn a robust, efficient representation from the start.

A landmark application is in Data-efficient Image Transformers (DeiT). DeiT introduced a distillation token—a learnable vector that attends to the teacher model's temperature-scaled class predictions throughout the transformer layers. The student model (DeiT) is trained with a combined loss: a standard cross-entropy loss with true labels and a distillation loss (e.g., KL divergence) against the softened outputs of a ConvNet teacher (like RegNet). This use of temperature scaling was pivotal in training competitive vision transformers without the massive, private datasets used to train the original ViT models.

Temperature scaling appears in advanced loss functions beyond standard KL divergence. In contrastive distillation, the InfoNCE loss commonly uses a temperature parameter τ to scale the similarity scores between positive and negative sample pairs in the representation space. This controls how concentrated the distribution is over similar examples. Similarly, in relational knowledge distillation, where the student learns to mimic relationships between data sample pairs as modeled by the teacher, temperature can soften the relationship distributions, making the relative similarities easier to learn. This demonstrates the parameter's utility in transferring structural, rather than just pointwise, knowledge.

Knowledge distillation is a model compression technique where a smaller, more efficient student model is trained to mimic the behavior and outputs of a larger, more powerful teacher model. The primary goal is to transfer the teacher's learned knowledge, including its generalization capabilities, into a compact form suitable for deployment in resource-constrained environments like mobile devices.

• Core Mechanism: The student is trained not just on the original hard labels (e.g., "this is a cat") but on the teacher's soft targets—the softened probability distribution over all classes.
• Key Benefit: Often, the distilled student model can match or even exceed the teacher's accuracy on the test set, despite having far fewer parameters, by learning the richer dark knowledge embedded in the teacher's outputs.

Soft targets are the probability distributions output by a teacher model's final softmax layer, especially when temperature scaling is applied. Unlike hard targets (one-hot encoded labels like [0, 0, 1, 0]), soft targets provide a richer training signal.

• Information Content: They contain the teacher's estimates of similarity between classes (e.g., a model might assign probabilities of 0.7 for "cat," 0.25 for "lynx," and 0.05 for "dog").
• Role in Distillation: The student model is trained using a distillation loss (often KL Divergence) to match these soft targets, learning the nuanced relationships between classes that the teacher has discovered.

Logits distillation is the most common form of knowledge distillation, where the student model is trained to directly match the logits (the raw, pre-softmax outputs) of the teacher model. This approach bypasses the softmax nonlinearity during the loss calculation, often leading to more stable gradients.

• Process: The teacher's logits are scaled by a temperature parameter (T), passed through a softmax to create soft targets, and then compared to the student's similarly scaled outputs.
• Loss Function: The primary objective is the Kullback-Leibler Divergence (KL Divergence) between the teacher and student's softened probability distributions, combined with a standard cross-entropy loss with the true labels.

Kullback-Leibler Divergence is a statistical measure of how one probability distribution diverges from a second, reference probability distribution. In knowledge distillation, it is the primary loss function for aligning the student's output distribution with the teacher's softened distribution.

• Mathematical Role: KL Divergence quantifies the information lost when the student's distribution Q is used to approximate the teacher's distribution P. Minimizing this loss forces the student to replicate the teacher's confidence patterns.
• Asymmetric Nature: It is not a distance metric (KL(P||Q) ≠ KL(Q||P)). In distillation, the teacher's distribution P is always the reference.

The teacher-student framework is the foundational architecture for knowledge distillation. It consists of two core components:

• Teacher Model: A large, pre-trained, and highly accurate model (often an ensemble or a very deep network) that serves as the source of knowledge.
• Student Model: A smaller, more efficient architecture (e.g., a shallower network or one with fewer channels) that is trained to replicate the teacher's behavior.

The framework defines the direction of knowledge transfer: from a fixed or static teacher to a trainable student. Variants like online distillation and mutual learning modify this static relationship.

Dark knowledge is a term coined by Geoffrey Hinton to describe the implicit, inter-class relationships learned by a trained neural network and embedded within its output probabilities. This knowledge is "dark" because it is not captured by the simple hard labels of the training data.

• Source: It emerges from the teacher model's softened output distribution. For example, a model trained on digits might show that a '3' has some visual similarity to an '8' and a '5', which is reflected in non-zero probabilities for those classes.
• Transfer Value: This relational information is what makes knowledge distillation powerful. By learning to mimic these softened outputs, the student model acquires a more robust and generalizable understanding than it would from hard labels alone.