Temperature Scaling

The temperature parameter (T) is a scalar value applied within the softmax function to control the sharpness of the output probability distribution. The modified softmax function for a logit vector z is: softmax(z_i) = exp(z_i / T) / Σ_j exp(z_j / T). When T = 1, it's the standard softmax. When T > 1, probabilities are 'softened,' making the distribution smoother and less confident. This reveals the dark knowledge—the relative similarities between classes—that the student model learns from.

The primary function of temperature scaling is to generate soft targets or soft labels. For example, in an image of a '7', a teacher might output logits leading to hard probabilities like [Cat: 0.01, Dog: 0.02, Seven: 0.95, Nine: 0.02]. With T=3, this becomes a softer distribution like [Cat: 0.05, Dog: 0.10, Seven: 0.70, Nine: 0.15]. This indicates to the student that a '7' is somewhat similar to a '9', but not to a 'cat'—information absent in the one-hot label. This richer signal is the core of effective distillation.

Temperature scaling is directly integrated into the distillation loss, typically the Kullback-Leibler (KL) Divergence Loss. The student is trained to minimize the divergence between its own softened outputs (using the same T) and the teacher's softened outputs. The total loss is often a weighted sum: L_total = α * L_CE(hard_labels) + (1-α) * T^2 * L_KL(teacher_soft_targets || student_soft_targets). The T^2 term is used to scale the gradients, as the softened probabilities produce smaller gradients magnitude as T increases.

Beyond distillation, temperature scaling (with T tuned on a validation set) is a premier post-hoc calibration technique. A well-calibrated model's predicted confidence aligns with its accuracy. A model might be accurate but overconfident (e.g., predicting 0.99 probability when it's only correct 80% of the time). Applying T > 1 can smooth its output distribution, reducing confidence on average and improving metrics like Expected Calibration Error (ECE) without retraining the model's weights.

Temperature scaling is often used in conjunction with other distillation paradigms:

• Logit-based Distillation: It is the foundational technique here.
• Feature-based Distillation (e.g., Attention Transfer): Temperature is not typically applied to intermediate features.
• Online Distillation: The temperature parameter can be applied to the peer models' outputs.
• Quantization-Aware Distillation (QAD): Temperature-scaled soft targets can help the student model learn a more robust representation before quantization. It is a complementary, not competing, technique.

Choosing the right temperature is critical. Common practices include:

• Typical Range: T is usually between 1 and 20, with 3-5 being a common starting point for vision and NLP tasks.
• Tuning: T is a hyperparameter validated on a downstream task's performance. Too low (≈1) provides little dark knowledge. Too high flattens the distribution excessively, losing necessary discriminative signal.
• Inference: The temperature is only used during the student's training. During final inference, the student model uses the standard softmax (T=1). The teacher's role, and thus temperature scaling, is complete.

Knowledge Distillation (KD) is the overarching model compression technique where a smaller student model is trained to mimic the predictive behavior of a larger, pre-trained teacher model. The primary mechanism is for the student to learn from the teacher's softened output probabilities (soft targets), which contain richer inter-class similarity information (dark knowledge) than hard one-hot labels. This process enables the deployment of high-accuracy models in resource-constrained environments.

• Core Objective: Transfer 'dark knowledge' from teacher to student.
• Key Components: Teacher model, student model, distillation loss function.
• Primary Use Case: Model compression for efficient inference.

Soft targets, or soft labels, are the probability distributions output by a teacher model's final softmax layer (often with temperature scaling applied). Unlike hard one-hot labels (e.g., [0, 0, 1, 0]), soft targets (e.g., [0.01, 0.04, 0.9, 0.05]) encode the teacher's relative confidence across all classes, capturing dark knowledge about inter-class similarities. These are the primary learning signal for the student model in knowledge distillation.

• Contain Dark Knowledge: Reveal which classes the teacher considers 'similar'.
• Generated via Softmax: Applied to the teacher's logits.
• Critical for KD: Student minimizes the difference between its outputs and these soft targets.

Kullback-Leibler Divergence Loss is the standard mimicry loss function used in knowledge distillation. It measures the statistical distance between the softened output probability distributions of the teacher model and the student model. The student is trained to minimize this divergence, forcing its predictions to align with the richer probability distribution of the teacher. It is typically combined with the standard cross-entropy loss in a weighted sum to form the total distillation loss.

• Measures Distribution Similarity: Quantifies how one probability distribution differs from another.
• Core of Distillation Loss: The term that enforces mimicry of the teacher's soft targets.
• Asymmetric: KL(P_teacher || P_student) is not the same as KL(P_student || P_teacher); the former is standard.

In the context of knowledge distillation, logits are the unnormalized, real-valued output vectors from the final linear layer of a neural network, preceding the softmax activation. They represent the model's raw scores for each class. During distillation, the teacher model's logits are scaled by a temperature parameter (T > 1) before being passed through the softmax function to produce the soft targets. The student model is trained to produce logits that, when similarly scaled and softened, match the teacher's distribution.

• Raw Model Outputs: Pre-softmax classification scores.
• Input to Softmax: Transformed into probabilities.
• Scaled by Temperature: T > 1 smoothes the resulting probability distribution for distillation.

A teacher model is a large, pre-trained, and typically high-accuracy neural network (e.g., BERT, ResNet-50) whose knowledge is transferred to a smaller student model during knowledge distillation. The teacher acts as a source of dark knowledge, providing soft targets from its output logits and, in feature-based distillation, intermediate feature representations or attention maps. The teacher is usually frozen during the distillation process.

• Source of Knowledge: Provides the training signal for the student.
• Often a SOTA Model: Chosen for its high performance on the target task.
• Frozen Parameters: Its weights are typically not updated during student training.

A student model is a smaller, more efficient neural network (e.g., DistilBERT, TinyBERT) that is trained to replicate the performance of a larger teacher model through knowledge distillation. The student's architecture is designed for lower computational and memory footprint, enabling faster inference. It learns by minimizing a distillation loss that aligns its softened outputs with the teacher's soft targets, often while also minimizing standard cross-entropy loss with ground-truth labels.

• Target of Compression: The model intended for efficient deployment.
• Learns via Mimicry: Trained on teacher soft targets, not just hard labels.
• Optimized for Efficiency: Has fewer parameters, layers, or lower precision than the teacher.