Distillation Loss

The total distillation loss is a weighted sum of two distinct components:

• Distillation Loss (Soft Loss): The Kullback-Leibler divergence between the softened probability distributions of the teacher and student.
• Student Loss (Hard Loss): The standard cross-entropy loss between the student's predictions and the ground-truth labels. The balance is controlled by a hyperparameter α (alpha), allowing practitioners to tune the relative importance of mimicking the teacher versus fitting the original data.

The KL divergence term measures how one probability distribution diverges from a reference distribution. In distillation:

• It is computed as KL(p_teacher || p_student), where both distributions are softened by a temperature parameter T > 1.
• Unlike cross-entropy with hard labels, KL divergence captures the relative probabilities of incorrect classes, transferring the teacher's knowledge about class similarities.
• Minimizing this term forces the student to replicate not just the teacher's top-1 prediction, but its entire output distribution.

Temperature T is applied identically to both teacher and student logits before computing the soft loss:

• High T (e.g., T=20): Produces a softer probability distribution, revealing subtle inter-class relationships and providing more informative gradients.
• Low T (e.g., T=1): Approaches standard softmax, focusing primarily on the predicted class.
• The distillation loss is typically scaled by T² to maintain gradient magnitudes, as higher temperatures proportionally reduce the gradient of the softmax function.

The distillation loss provides a richer training signal than standard supervision:

• Standard cross-entropy provides a gradient that only pushes the student toward the correct class.
• KL divergence with soft targets provides gradients that push the student toward the teacher's full distribution, including negative gradients on incorrect classes proportional to the teacher's confidence.
• This results in lower variance gradients and more efficient learning, as the student receives information about the relative plausibility of all classes from every training example.

The α hyperparameter determines the trade-off between imitation and accuracy:

• α = 0: Pure student loss; the model ignores the teacher entirely.
• α = 1: Pure distillation loss; the model only mimics the teacher's outputs.
• Typical setting: α between 0.1 and 0.5, giving more weight to the distillation loss when the teacher is significantly more accurate than a student trained from scratch.
• Adaptive weighting: Some implementations dynamically adjust α during training, increasing the hard loss weight as the student converges.

The distillation loss is the mathematical mechanism that extracts dark knowledge from the teacher:

• Dark knowledge is the information encoded in the relative probabilities assigned to incorrect classes.
• For example, a teacher trained on ImageNet might assign a higher probability to 'ambulance' than 'garbage truck' for an image of a 'minivan', revealing learned similarity structures.
• The KL divergence term captures this by penalizing the student when its distribution over all classes diverges from the teacher's, transferring structural knowledge that is absent from hard labels.

The primary component of distillation loss that measures how the student's predicted probability distribution diverges from the teacher's soft targets. It is an asymmetric metric, quantifying the information lost when the student distribution is used to approximate the teacher distribution. Minimizing the KL divergence forces the student to replicate not just the correct class prediction, but the full dark knowledge of relative probabilities among all incorrect classes.

A critical hyperparameter applied within the softmax function that controls the entropy of the probability distributions used in the KL divergence calculation. A higher temperature (T > 1) produces softer targets, amplifying the importance of the teacher's dark knowledge. The same temperature must be used for both the teacher and student during training to ensure the gradients are properly scaled, though it is reset to 1 for standard inference.

The second, often weighted, component of the total distillation loss. This is the standard supervised loss computed between the student's hard predictions and the ground-truth labels. Including this term anchors the student to the actual task objective and prevents it from solely mimicking a potentially imperfect teacher. The balance between this term and the KL divergence is controlled by a mixing coefficient, typically denoted as alpha.

The implicit generalization knowledge encoded in the teacher's softmax output, specifically in the probabilities assigned to incorrect classes. For example, an image of a 'cat' might have a much higher probability for 'dog' than for 'truck'. This relative structure reveals the teacher's learned similarity manifold. Distillation loss, via KL divergence on soft targets, is the mechanism for transferring this non-obvious, valuable information to the student.

A student model whose performance is measured by its fidelity—the degree to which its predictions match the teacher's on unseen data—rather than solely by accuracy against ground-truth labels. The distillation loss function directly optimizes for this metric. High fidelity indicates the student has successfully internalized the teacher's decision boundaries and generalization patterns, making it a faithful surrogate for explanation purposes.

An extension of standard distillation loss that adds a penalty term for mismatches between the intermediate feature representations of the teacher and student. Instead of only matching final logits, the student is trained to minimize the L2 distance between its hidden layer activations and those of the teacher. This is particularly useful when the student's architecture is too small to capture the full output distribution from logits alone.