Temperature Scaling

The temperature parameter (T) is a scalar value that divides the logits (pre-softmax activations) before applying the softmax function: softmax(logits / T). It directly controls the entropy of the output distribution.

• T = 1: Standard softmax, producing the model's native confidence distribution.
• T > 1: 'Softens' the distribution, increasing entropy. Probabilities become more uniform, revealing the model's relative confidence across classes—the so-called dark knowledge.
• T < 1: 'Hardens' the distribution, making the winning class's probability approach 1.0. This is used for inference-time confidence calibration or creating sharper targets.

The primary role of temperature scaling is to generate soft targets for knowledge distillation. A high temperature (e.g., T=3 to T=10) is applied to the teacher model's logits.

Key Mechanism:

• The softened probabilities preserve the teacher's knowledge of inter-class relationships. For example, an image of a 'cat' might have a high probability for 'cat', a moderate probability for 'lynx', and a near-zero probability for 'car'. This richer signal teaches the student model more than a one-hot 'cat' label.
• The student is trained using a loss function (like KL Divergence Loss) to match this softened distribution, often while also learning from the true hard labels.

Temperature scaling is a state-of-the-art method for post-hoc calibration of neural networks. A model is calibrated when its predicted confidence (e.g., softmax probability of 0.9) matches its actual accuracy (e.g., is correct 90% of the time).

Process:

1. After training, a separate validation set is used to find a single scalar temperature T.
2. This T (often < 1) is applied to the model's logits at inference time.
3. It adjusts the output probabilities to better reflect true likelihoods without changing the model's predicted class ranking (argmax).

• This is critical for risk-sensitive applications like medical diagnosis or autonomous driving.

The operation is defined by a simple modification to the standard softmax:

softmax(z_i, T) = exp(z_i / T) / Σ_j exp(z_j / T)

Where z are the logits and T is the temperature.

Derivative & Gradient Effect: During distillation training, the gradient passed to the student with respect to its logits s_i is: ∂L_KL / ∂s_i = (1/T^2) * (q_i - p_i) where q is the student's softened distribution and p is the teacher's softened distribution.

• The 1/T^2 factor acts as a gradient scaling term. Higher temperatures produce smaller gradients, effectively smoothing the loss landscape and stabilizing training.

Temperature scaling is a key component in distillation-based continual learning algorithms like Learning without Forgetting (LwF).

Application:

• When learning a new task, the model's predictions on new data for old tasks are recorded using a high temperature to create soft targets.
• These soft targets are then used as a regularization loss when training on the new task, penalizing the model for deviating from its previous knowledge.
• This technique helps mitigate catastrophic forgetting by preserving the 'dark knowledge' of how previous classes relate to each other, not just their hard labels.

Typical Workflows:

• For Distillation: Use a high T (e.g., 3-10) on the teacher. Train the student with a weighted sum of distillation loss (KL Divergence at temperature T) and standard cross-entropy loss (at T=1).
• For Calibration: Train model normally. On a held-out validation set, optimize T > 0 to minimize Negative Log Likelihood (NLL) or Expected Calibration Error (ECE). Apply this T at test time.

Considerations:

• Temperature is a hyperparameter and must be tuned for the specific model and dataset.
• It does not change the model's architecture or the ranking of its predictions.
• The technique is computationally inexpensive, adding negligible overhead.

Temperature scaling is a post-hoc calibration method applied after a model is trained. It adjusts the confidence of a model's predictions to better match its actual accuracy. A well-calibrated model that predicts a class with 80% probability should be correct 80% of the time. By tuning a single temperature parameter (T) on a validation set, it softens or sharpens the output distribution, reducing overconfidence without changing the model's predicted ranking of classes. This is critical for risk-sensitive applications like medical diagnosis or autonomous driving, where confidence scores must be meaningful.

This is the technique's namesake and most common application. Here, temperature scaling is applied to the teacher model's logits before the student model is trained. A temperature (T > 1) is used to create soft targets—a richer, smoother probability distribution that reveals the dark knowledge of class relationships learned by the teacher. For example, an image of a 'husky' might have high probability for 'dog' and a non-zero probability for 'wolf,' which is more informative than a one-hot 'husky' label. The student learns from these soft targets using a Kullback-Leibler (KL) Divergence loss, often in combination with the standard cross-entropy loss with true labels.

In continual learning, temperature scaling is used within algorithms like Learning without Forgetting (LwF). When learning a new task, the model's predictions on data from previous tasks are softened with a temperature parameter (T > 1) to create pseudo-soft labels. These labels are then used as distillation targets to regularize training on the new task, penalizing the model if it drifts too far from its old knowledge. This acts as a regularization constraint, helping to preserve performance on past tasks without requiring access to the original training data, thus directly addressing the plasticity-stability trade-off.

In self-supervised learning, particularly contrastive frameworks like SimCLR, a temperature parameter (τ) in the InfoNCE loss function controls the concentration of the similarity distribution. A properly tuned τ sharpens the model's ability to distinguish between positive and negative pairs in the embedding space. While conceptually similar, this 'temperature' operates on normalized embeddings (cosine similarity) rather than logits. It is crucial for learning high-quality representations, as it determines how harshly the model penalizes hard negative examples, directly impacting the separation and uniformity of the learned latent space.

In reinforcement learning, particularly in policy gradient methods, temperature scaling is applied to the action probability distribution output by the policy network. A higher temperature increases entropy, encouraging exploration by making the policy more stochastic. A lower temperature reduces entropy, making the policy more deterministic and exploitative of known high-reward actions. This parameter is often annealed during training, starting with high exploration (high T) and gradually shifting to exploitation (low T). It's a key hyperparameter in algorithms like Soft Actor-Critic (SAC), where it automatically adjusts to maximize expected reward while maintaining a minimum entropy.

When training with noisy or ambiguous labels, applying temperature scaling (T > 1) to the one-hot labels can create label smoothing. This technique converts hard, potentially incorrect labels into softer, more uncertain distributions (e.g., [0.9, 0.1] -> [0.7, 0.3]). It acts as a form of regularization, preventing the model from becoming overconfident in potentially wrong answers and improving generalization. While distinct from standard label smoothing (which uses a uniform distribution), temperature scaling on logits achieves a similar effect of reducing model overconfidence and can be more adaptive to the specific class relationships present in the data.

Knowledge distillation is a model compression and transfer learning technique where a smaller, more efficient student model is trained to mimic the behavior of a larger, more complex teacher model. The student learns not just from hard labels, but from the teacher's softened output probabilities or intermediate feature representations, transferring the teacher's generalized knowledge.

• Core Purpose: To create smaller, faster models that retain the performance of larger ones.
• Key Signal: The dark knowledge in the teacher's probability distribution.
• Primary Use Cases: Model deployment on edge devices, ensemble compression, and as a regularization method in continual learning.

Dark knowledge refers to the rich, non-obvious information contained within the softened probability distribution (logits) of a trained neural network. It reveals the teacher model's relative confidence and similarities between different classes, which is more informative than a simple one-hot label.

• Revealed by: Applying temperature scaling (T > 1) to the teacher's logits.
• Example: For an image of a '3', a teacher might output high probability for '3', but also non-zero probabilities for '8' and '5', indicating visual similarity. This relational information is the dark knowledge transferred to the student.

Soft targets are the probability distributions output by a teacher model after temperature scaling. They serve as the primary training signal for the student model in logit-based knowledge distillation.

• Contrast with Hard Targets: Hard targets are one-hot encoded labels (e.g., [0, 0, 1, 0]). Soft targets are probabilistic (e.g., [0.01, 0.05, 0.9, 0.04]).
• Advantage: Provide a richer gradient signal and convey inter-class relationships, leading to better generalization and calibration in the student model.

Kullback-Leibler (KL) Divergence Loss is the standard objective function used in logit-based knowledge distillation. It measures the statistical difference between the softened probability distributions of the teacher and student models.

• Mathematical Role: Minimizing KL divergence forces the student's output distribution to match the teacher's.
• Typical Formulation: The total loss is often a weighted sum of the KL divergence loss (for soft targets) and the standard cross-entropy loss (for hard labels).
• Direct Application: Central to algorithms like Learning without Forgetting (LwF) for mitigating catastrophic forgetting.

Learning without Forgetting is a seminal continual learning algorithm that leverages knowledge distillation to prevent catastrophic forgetting. When learning a new task, the model uses its own predictions on the new data (before updating) as soft targets to preserve its performance on previous tasks.

• Mechanism: Applies a distillation loss between the model's current outputs and its outputs frozen from before learning the new task.
• Key Benefit: Does not require storing raw data from past tasks (exemplar-free).
• Foundation: Demonstrates the direct application of distillation as a regularization technique for stability in continual learning.

Catastrophic forgetting (or catastrophic interference) is the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when it is trained on new, different data. This is the core problem that continual learning systems aim to solve.

• Root Cause: Unconstrained plasticity in neural networks, where new learning overwrites old weight configurations.
• The Trade-off: Central to the plasticity-stability trade-off—a model must balance learning new information (plasticity) with retaining old knowledge (stability).
• Mitigation Role of Distillation: Techniques like LwF and distillation for forgetting use soft targets from a model's past state to constrain updates and protect old knowledge.