Temperature Scaling

The temperature parameter (T) is a scalar value applied within the softmax function to control the entropy of the output probability distribution. The modified softmax for logit (z_i) is: ( \sigma(z_i, T) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}} ).

• T = 1: Standard softmax, producing the original distribution.
• T > 1: 'Softens' the distribution, increasing probabilities for non-argmax classes and revealing dark knowledge about inter-class similarities.
• T < 1: 'Hardens' the distribution, making it more peaky and confident, approaching a one-hot label.

The primary function of temperature scaling (with T > 1) is to soften the teacher's output probabilities. This transforms hard, one-hot-like labels into a richer supervisory signal.

• Example: For a cat/dog/horse image, a teacher's raw logits might yield probabilities [0.9, 0.09, 0.01]. With T=2, this could soften to [0.7, 0.2, 0.1].
• This softened distribution encodes relational information (e.g., 'cat' is more similar to 'dog' than to 'horse'), which is the dark knowledge crucial for effective distillation.
• The student learns from these nuanced probabilities, leading to better generalization than learning from hard labels alone.

Temperature scaling is integral to the standard distillation loss, which combines a cross-entropy loss with the Kullback-Leibler (KL) Divergence between the teacher and student's softened outputs.

• The total loss is often: ( L = (1 - \alpha) L_{CE}(y, \sigma(z_s)) + \alpha T^2 L_{KL}(\sigma(z_t/T), \sigma(z_s/T)) ).
• The ( T^2 ) factor compensates for the gradient scaling introduced by the temperature.
• The hyperparameter ( \alpha ) balances learning from true labels versus learning from the teacher's softened distribution.

Beyond distillation, temperature scaling is a primary method for model calibration. A well-calibrated model's predicted confidence aligns with its actual accuracy.

• A single scalar T is learned on a validation set to scale the logits post-training.
• This simple linear operation can significantly reduce overconfidence (where softmax probabilities are excessively peaky) without changing the model's argmax predictions.
• It is a key component in Platt scaling for neural networks, providing a lightweight way to improve probability estimates for downstream decision-making.

During distillation, the same temperature T is applied to both teacher and student logits for the KL divergence term. This ensures a consistent, smooth target for the student to match.

• High T (e.g., 10): Creates a very uniform, high-entropy target. This can be beneficial in early training to prevent the student from overfitting to the teacher's potentially noisy argmax.
• Optimal T: Is task and dataset-dependent. A common heuristic is to start with a higher T (e.g., 4-10) and potentially anneal it during training, or to tune it as a hyperparameter.
• At inference, the student uses standard softmax (T=1). The temperature is purely a training-time mechanism.

While powerful, temperature scaling has specific constraints and trade-offs.

• Single-Scalar Limitation: It applies a uniform transformation to all logits, which may not be optimal for all samples or classes. More advanced methods like vector scaling or matrix scaling exist but add complexity.
• Hyperparameter Tuning: The optimal value of T must be searched for, typically via validation performance. An incorrect T can provide less informative targets than hard labels.
• Not a Panacea: It primarily aids in transferring dark knowledge via output logits. For complex student architectures, it is often combined with feature mimicking or attention transfer from intermediate layers for greater knowledge fidelity.

Logits Distillation is the foundational technique where the student model is trained to directly match the raw logits (pre-softmax outputs) of the teacher model. Temperature Scaling is applied within this process to soften the teacher's output distribution.

• Core Mechanism: Uses a distillation loss (often Kullback-Leibler Divergence) between the temperature-scaled logits of teacher and student.
• Objective: Transfers the teacher's full scoring rationale, including its relative confidence across all classes, not just the top prediction.

Soft Targets are the probability distributions produced by applying the softmax function to the teacher's temperature-scaled logits. They are the supervisory signal in logits distillation.

• Contrast with Hard Labels: Unlike one-hot encoded labels, soft targets provide probabilistic and richer supervisory information.
• Dark Knowledge: They encode the teacher's learned inter-class relationships (e.g., that a 'cat' is more similar to a 'lynx' than to a 'truck'). This relational information is the 'dark knowledge' transferred to the student.

Kullback-Leibler (KL) Divergence Loss is the primary objective function used in logits distillation to measure the difference between the soft target distribution from the teacher and the soft prediction distribution from the student.

• Mathematical Role: It quantifies how much information is lost when the student's distribution is used to approximate the teacher's. Minimizing this loss aligns the two distributions.
• Combined Loss: In practice, the total loss is often a weighted sum: L_total = α * L_CE(hard_labels) + (1-α) * T^2 * L_KL(soft_targets). The T^2 term counteracts the scaling of gradients introduced by the temperature parameter.

The Teacher-Student Framework is the overarching paradigm for knowledge distillation. A large, accurate, and often cumbersome teacher model provides training guidance to a smaller, efficient student model.

• Static vs. Online: In standard distillation, the teacher is pre-trained and frozen. In online distillation, both models are trained concurrently.
• Architecture Independence: The student's architecture is typically different (and smaller) than the teacher's, enabling model compression and efficiency gains.

Dark Knowledge refers to the implicit, relational information learned by a trained model that is not contained in the final hard class label. It is the primary knowledge transferred via soft targets in distillation.

• Source of Information: It resides in the relative magnitudes of the logits for non-true classes. A high score for a 'sedan' when the input is a 'truck' reveals learned similarity.
• Value Proposition: This knowledge helps the student model generalize better and often achieve higher accuracy than if trained on hard labels alone, as it learns a smoother, more informed function.

The Temperature Parameter (T) is the scalar hyperparameter central to Temperature Scaling. It controls the 'softness' of the probability distribution output by the softmax function.

• Effect on Softmax: As T > 1, the output distribution becomes softer (more uniform), amplifying the differences in dark knowledge. As T → ∞, the distribution becomes uniform. As T = 1, it's the standard softmax. As T < 1, it becomes sharper, approaching a one-hot distribution.
• Optimization Role: A higher T produces smaller gradients for the distillation loss, leading to smoother optimization. The optimal T is task and dataset-dependent and is a key hyperparameter to tune.