Knowledge Distillation

The student model is trained on the teacher's softened probability distribution over classes, not just the ground-truth hard label. A temperature parameter (T) in the final softmax layer controls the softness of these targets. Higher temperatures reveal the dark knowledge encoded in the teacher's inter-class similarities—for example, a teacher classifying a car image might assign small probabilities to 'truck' and 'van', teaching the student about visual hierarchies that one-hot labels cannot convey.

The training objective combines two loss terms:

• Distillation loss: Kullback-Leibler divergence between the softened student and teacher outputs, scaled by T²
• Student loss: Standard cross-entropy between the student's hard predictions and ground-truth labels A weighting hyperparameter α balances these terms. The T² scaling ensures the gradient magnitudes from soft targets remain comparable when temperature is adjusted, preventing the distillation signal from vanishing at high temperatures.

Knowledge distillation enables dramatic model compression while preserving accuracy:

• BERT-base (110M parameters) distilled into DistilBERT (66M parameters) retains 97% of performance
• ResNet-50 teacher can train a MobileNet student with 10× fewer FLOPs
• GPT-3 (175B) to GPT-3-distilled variants achieving 50× size reduction The student architecture is unconstrained—it can be a shallower network, a quantized version, or an entirely different architecture family optimized for the target hardware.

Beyond output probabilities, the student can learn from the teacher's intermediate representations. Techniques include:

• Hint-based training: Matching feature maps at specific intermediate layers using L2 loss
• Attention transfer: Aligning spatial attention maps between teacher and student
• Relation-based distillation: Preserving pairwise distance relationships between samples in the teacher's embedding space This is particularly effective when the student and teacher have different output spaces or when training for dense prediction tasks like semantic segmentation.

Offline distillation pre-trains the teacher first, then freezes it during student training—the standard approach requiring a high-capacity teacher checkpoint. Online distillation trains both models simultaneously, with the teacher and student co-evolving. Self-distillation eliminates the teacher entirely: the student's own predictions from previous epochs serve as soft targets, iteratively refining its own knowledge without any external model. This is valuable when compute budgets preclude training a large teacher.

Knowledge distillation directly addresses the accuracy-efficiency trade-off in manufacturing edge AI:

• A distilled defect detection model running on an NPU-accelerated smart camera achieves sub-10ms inference with teacher-equivalent accuracy
• Distilled models fit within the tight memory budgets of microcontrollers and embedded Linux devices
• The technique preserves the teacher's out-of-distribution detection capability, critical for flagging novel defect types on the factory floor
• Combined with post-training quantization, distilled models achieve 4× further compression without additional accuracy loss

The foundational two-model training paradigm where a large, high-capacity teacher model generates soft probability distributions over output classes, and a compact student model is trained to mimic these distributions rather than hard ground-truth labels. The teacher's softened outputs—controlled by a temperature parameter in the softmax function—reveal inter-class similarities and dark knowledge that encode richer representational information than one-hot labels alone. The student minimizes the Kullback-Leibler divergence between its own softened outputs and the teacher's, often combined with a standard cross-entropy loss against true labels.

The implicit, non-obvious information encoded in a teacher model's soft probability distribution that goes beyond simple class labels. When a teacher classifies an image, it assigns small but non-zero probabilities to incorrect classes—revealing that a cat image shares visual features with dogs, or that a handwritten '3' resembles an '8'. This relative similarity structure provides a richer training signal than hard labels, enabling the student to generalize better from fewer examples by learning the teacher's internal error landscape and class relationships.

Hard labels are discrete one-hot encoded class assignments—a sample is definitively 'cat' or 'dog'. Soft targets are continuous probability vectors produced by the teacher, where a sample might be 0.85 cat, 0.12 dog, 0.02 fox, and 0.01 other. Training on soft targets provides per-sample loss weighting that emphasizes difficult or ambiguous examples and encodes the teacher's uncertainty. The student learns not just correct answers but the manifold geometry of the teacher's decision boundaries, often achieving higher accuracy than training on hard labels alone.

A softmax modification that controls the entropy of the teacher's output distribution. The standard softmax divides logits by a temperature parameter T: higher values (T > 1) produce softer, more uniform probability distributions that expose more dark knowledge; T = 1 recovers the standard softmax; T → 0 approaches hard argmax. During distillation, the teacher and student use the same elevated temperature, but the student's loss is scaled by T² to maintain gradient magnitudes. Typical values range from 2 to 20, tuned based on the teacher's confidence calibration.

The student model is optimized using a composite loss that balances multiple objectives:

• Distillation loss: KL divergence between teacher and student softmax outputs at temperature T, weighted by α
• Student loss: Standard cross-entropy against ground-truth hard labels, weighted by (1 - α)
• Feature-based loss (optional): L2 distance between intermediate layer activations, aligning internal representations
• Relation-based loss (optional): Preserving pairwise distance relationships between samples in the teacher's embedding space The α hyperparameter typically ranges from 0.5 to 0.9, heavily favoring the teacher's guidance.

A variant where the teacher and student share identical architectures, with the student training on soft targets from a previously trained or concurrently training copy. This technique acts as a regularizer, reducing overfitting by forcing the model to produce consistent predictions across training epochs. Self-distillation can be applied iteratively—each generation's student becomes the next generation's teacher—often yielding progressive accuracy improvements without increasing model size. It is particularly effective when labeled data is scarce and the model benefits from its own softened predictions.