Knowledge Distillation

The fundamental framework involves a pre-trained, high-capacity teacher model and an untrained, lightweight student model. The student learns not from the original ground-truth labels alone, but by mimicking the teacher's behavior. The teacher provides soft targets—probability distributions over classes—which contain richer information than hard labels, including inter-class similarities and the teacher's uncertainty estimates.

A critical mechanism where the teacher's output logits are divided by a temperature parameter (T) before applying softmax. Higher temperatures produce softer probability distributions, revealing the dark knowledge of non-target classes. The student is trained to minimize the divergence between its own softened outputs and the teacher's softened outputs using Kullback-Leibler (KL) divergence loss, often combined with a standard cross-entropy loss against ground truth.

The training objective combines two loss components:

• Distillation Loss: KL divergence between the student's softened logits and the teacher's softened logits, weighted by T².
• Student Loss: Standard cross-entropy between the student's hard predictions and the true labels. A hyperparameter α balances these two terms. This dual objective ensures the student learns both the teacher's generalization patterns and the ground-truth signal.

Beyond matching final outputs, the student can learn from the teacher's intermediate representations. The student is trained to minimize the distance between its own hidden layer activations and the teacher's corresponding feature maps. Techniques include:

• Hint-based training: Aligning feature maps of specific layers.
• Attention transfer: Matching spatial attention maps.
• Relation-based distillation: Preserving relationships between data samples in the feature space.

Variants that eliminate the need for a separate pre-trained teacher:

• Online distillation: Teacher and student are trained simultaneously, with the teacher updating from the student's performance.
• Self-distillation: A model acts as its own teacher; deeper layers distill knowledge to shallower layers, or an ensemble of sub-networks within a single model teaches the full network. This improves accuracy without any external teacher model.

Knowledge distillation is widely deployed to compress large models for production:

• BERT → DistilBERT: A 40% smaller model retaining 97% of BERT's language understanding.
• ViT → MobileViT: Distilling vision transformers into mobile-friendly architectures.
• LLM distillation: Compressing models like GPT-4 into smaller, task-specific variants for low-latency inference.
• Ensemble distillation: Compressing an ensemble of diverse models into a single student that captures their collective wisdom.

The foundational framework where a large, high-capacity teacher model generates soft labels—probability distributions over classes—that a compact student model learns to mimic. Unlike hard labels, soft labels capture the teacher's uncertainty and inter-class relationships, providing richer supervisory signals. The student is trained to minimize the Kullback-Leibler divergence between its softened output distribution and the teacher's, often using a temperature parameter to control distribution smoothness.

A hyperparameter T applied to the softmax function that controls the softness of probability distributions during distillation. Higher temperatures (e.g., T=20) produce softer distributions that reveal more information about the teacher's internal representations, exposing subtle class similarities. The same temperature is used for both teacher and student during training, but set to T=1 during inference. This mechanism is critical for transferring dark knowledge—the hidden patterns in the teacher's output that go beyond simple class predictions.

The training objective combines multiple loss components:

• Distillation loss: KL divergence between softened teacher and student outputs
• Student loss: Standard cross-entropy with ground-truth hard labels
• A weighted sum balances mimicking the teacher versus learning from data This dual-objective approach ensures the student both replicates the teacher's nuanced behavior and maintains accuracy on the original task. The mixing ratio is a tunable hyperparameter, typically giving more weight to distillation loss.

An extension beyond output-level distillation where the student learns to replicate the teacher's intermediate representations. The student is trained to minimize the distance between its hidden layer activations and those of the teacher, often using hint layers and guided layers. This transfers not just what the teacher predicts, but how it internally processes information. Techniques include attention transfer, where the student mimics the teacher's attention maps, and neuron selectivity transfer using maximum mean discrepancy.

A variant where the teacher and student share the same architecture, with the student being an earlier checkpoint or a copy of the model. The model is trained iteratively: a trained model becomes the teacher for a new instance of itself. This process can improve generalization without any architectural change. Self-distillation has been shown to reduce overfitting and improve calibration, effectively allowing a model to learn from its own softened predictions in a bootstrapping fashion.

A dynamic paradigm where teacher and student are trained simultaneously rather than sequentially. Both models learn together, with the teacher updating its parameters alongside the student. This eliminates the need for a pre-trained teacher and allows mutual learning. Common implementations include deep mutual learning, where multiple peer networks teach each other, and co-distillation, where models of the same architecture exchange knowledge during training.