Knowledge Distillation

The core framework consists of a pre-trained, high-capacity teacher model and a lightweight student model. The student is trained not on the original hard labels, but on the teacher's soft targets—the full probability distribution over all classes. This distribution encodes rich dark knowledge about inter-class similarities that one-hot labels discard.

A temperature parameter (T) is applied to the softmax function to soften the teacher's output distribution:

• High T (T > 1): Produces a smoother probability distribution, revealing subtle relationships between classes
• Low T (T = 1): Approaches standard hard-label classification
• The student is trained with the same high temperature, but evaluated at T=1
• Typical values range from 2 to 20, tuned as a hyperparameter

The student model is optimized using a weighted combination of two loss terms:

• Distillation loss: Kullback-Leibler divergence between the softened student and teacher outputs, scaled by T²
• Student loss: Standard cross-entropy between student predictions and ground-truth hard labels
• The weighting hyperparameter α balances imitation versus ground-truth fidelity
• This dual objective ensures the student learns both the teacher's generalization patterns and the correct classifications

The teacher's soft targets contain dark knowledge—information about the relative probabilities of incorrect classes. For example, when classifying a handwritten '3', the teacher might assign higher probability to '8' than to 'dog', encoding structural similarity. This similarity knowledge:

• Provides richer gradients per training example
• Acts as an implicit regularizer
• Allows the student to generalize better than training on hard labels alone
• Is the key mechanism behind distillation's effectiveness

Three primary distillation paradigms exist:

• Offline distillation: A pre-trained teacher is frozen; the student trains on its outputs. Most common and straightforward
• Online distillation: Teacher and student are trained simultaneously, with the teacher continuously updating. Useful when no suitable pre-trained teacher exists
• Self-distillation: The student and teacher share the same architecture. A previously trained version of the model serves as its own teacher, improving performance without external models

Knowledge distillation extends beyond image classification to diverse domains:

• Object detection: Distilling bounding box regression and region proposal networks
• Natural language processing: Compressing large language models like BERT into TinyBERT for on-device inference
• Speech recognition: Transferring acoustic models to lightweight architectures
• Generative models: Distilling diffusion models to reduce sampling steps
• Multi-task distillation: A single teacher trained on multiple tasks distills into a unified multi-task student

A technique that reduces the numerical precision of a network's parameters from high-bit floating-point (FP32) to low-bit integers (INT8 or INT4). This dramatically accelerates inference and shrinks memory footprint with minimal accuracy loss.

• Post-training quantization converts a pre-trained model without retraining
• Quantization-aware training simulates low-precision effects during training for higher fidelity
• When paired with distillation, the student can be trained to be robust to quantization noise from the start

The foundational framework for knowledge distillation. A large, high-capacity teacher model generates soft probability distributions over classes, which a compact student model is trained to replicate. The student learns not just the correct answer, but the teacher's confidence across all possible outputs.

• The soft targets encode dark knowledge about inter-class similarities
• A temperature parameter in the softmax controls the smoothness of the probability distribution
• The student can be a smaller network, a pruned version, or a quantized variant of the teacher

A hybrid paradigm that combines federated learning with knowledge distillation. Instead of sharing model weights, clients exchange soft predictions on a public or synthetically generated dataset. This preserves data privacy while enabling collaborative model improvement across factory fleets.

• Clients train local student models on private data and share only aggregated outputs
• A central teacher is updated from the collective knowledge without raw data exposure
• Particularly valuable when clients have heterogeneous model architectures or compute capabilities

The tendency of a neural network to abruptly overwrite previously learned knowledge when trained on new data. In the context of distillation, a well-trained teacher can serve as a stable reference to mitigate forgetting in a continuously learning student.

• The teacher's soft outputs act as a regularizing signal, preserving old task boundaries
• Elastic weight consolidation and replay buffers are complementary techniques
• Critical for factory models that must adapt to new product lines without losing defect detection on legacy products