Knowledge Distillation

The foundational framework of knowledge distillation. A high-capacity teacher model (often an ensemble or large transformer) generates soft probability distributions over outputs. A compact student model is trained not just on hard labels, but to match these soft targets, which encode rich inter-class similarity structures that hard labels discard. The student learns the teacher's generalization patterns, often outperforming a model trained on raw data alone.

A critical hyperparameter in the distillation loss function. A temperature parameter (T) is applied to the softmax of both teacher and student logits before computing the KL divergence. Higher temperatures (e.g., T=20) soften the probability distribution, revealing the dark knowledge of which classes the teacher considers similar. This prevents the student from overfitting to the teacher's overconfident predictions on easy examples.

A complementary compression technique that removes individual weights or entire neurons from a trained network based on a saliency criterion, such as magnitude. Unstructured pruning zeroes out individual weights, leading to sparse matrices, while structured pruning removes entire channels or layers, yielding immediate inference speedups on commodity hardware without specialized sparse compute libraries.

Reduces model footprint by converting 32-bit floating-point weights and activations to lower-precision formats like INT8 or FP16. This technique is often applied to a distilled student model for a second stage of compression. Quantization-aware training (QAT) simulates quantization noise during distillation, while post-training quantization (PTQ) requires only a small calibration dataset, making it ideal for edge deployment.

An extension beyond matching final output probabilities. The student is trained to mimic the intermediate representations of the teacher, such as feature maps in convolutional layers or attention patterns in transformers. A regression loss (L2 or cosine similarity) aligns the student's hidden states with the teacher's, transferring the hierarchical feature extraction process directly. This is especially effective when the student and teacher have similar depth but different widths.

A technique where the teacher is an ensemble of diverse models. The student learns to mimic the averaged soft predictions of the entire committee, effectively compressing the collective wisdom of multiple high-capacity models into a single fast one. This captures model uncertainty and often yields a student that generalizes better than any single teacher, at a fraction of the ensemble's inference cost.