Knowledge Distillation

The foundational two-model framework where a large, pre-trained teacher generates soft targets for a compact student to mimic.

• Teacher Model: High-capacity, often an ensemble, frozen during distillation.
• Student Model: Lightweight architecture designed for low-latency inference.
• Capacity Gap: The student learns not just the final answer but the teacher's nuanced output distribution.

Instead of training on hard one-hot labels, the student learns from the teacher's softened probability distribution over all classes.

• Temperature (T): A hyperparameter applied to the softmax function to control output smoothness.
• High T: Produces softer probabilities, revealing inter-class similarities learned by the teacher.
• Dark Knowledge: The rich information about class relationships contained in the teacher's soft targets.

The student is optimized using a composite loss that balances mimicking the teacher with learning from ground truth.

• Kullback-Leibler Divergence: Measures the difference between the student's softened output and the teacher's soft targets.
• Cross-Entropy Loss: Standard supervised loss between the student's hard predictions and true labels.
• Linear Combination: Total loss is typically a weighted sum: L = α * L_CE + (1-α) * L_KD.

The most common paradigm where the student mimics the final output layer of the teacher.

• Logit Matching: Directly regressing the student's pre-softmax logits to match the teacher's.
• Applicable Domains: Effective for image classification, language model fine-tuning, and speech recognition.
• Limitation: Ignores the rich intermediate representations learned by the teacher's hidden layers.

The student learns to replicate the intermediate feature maps or hidden states of the teacher, not just the final output.

• Hint Layers: Selected intermediate layers from the teacher used as supervision targets.
• Projection Layers: Learned transformations to align student and teacher feature dimensions.
• Benefit: Transfers structural and representational knowledge, improving generalization on complex tasks.

Transfers the mutual relationships between data samples or feature maps, preserving the teacher's learned manifold structure.

• Instance Relationship Graph: Distills the similarity matrix between samples in a batch.
• Flow of Solution Procedure (FSP): Captures the interaction between two layers via a Gram matrix.
• Goal: Enforce structural consistency in the embedding space, not just point-wise accuracy.

The foundational framework where a large, high-capacity teacher model generates soft targets—probability distributions over classes—that a compact student model is trained to mimic. Unlike hard label training, the student learns the teacher's dark knowledge: the relative probabilities of incorrect classes, which encode rich similarity structures. The student minimizes the Kullback-Leibler divergence between its softened output distribution and the teacher's, using a temperature parameter to control softness.

A compression technique that removes individual weights or entire neurons from a trained network based on a saliency criterion. Unstructured pruning zeros out individual weights, producing sparse matrices that require specialized hardware for speedup. Structured pruning removes entire channels or attention heads, yielding dense sub-networks that run efficiently on commodity hardware. Often combined with distillation by using the pruned model as a student initialization.

Reduces model precision from 32-bit floating point to lower bit-widths like INT8 or INT4 without retraining. Quantization maps continuous weight values to discrete bins, dramatically shrinking memory footprint and enabling integer-arithmetic inference. When paired with distillation, a student trained on soft targets often exhibits greater robustness to quantization error than one trained on hard labels alone, a technique known as quantization-aware distillation.

Extends standard distillation beyond output probabilities by forcing the student to match the teacher's intermediate representations. The student minimizes the distance between its hidden layer activations and the teacher's, often using mean squared error or contrastive losses. This transfers representational knowledge at multiple abstraction levels. Common variants include FitNet hints-based training and attention transfer, where the student mimics the teacher's self-attention maps.

A paradigm where the teacher and student share the same architecture, with the student being an earlier checkpoint or a re-initialized copy of the same network. The model distills its own knowledge, often using predictions from later training epochs to regularize earlier ones. This eliminates the need for a separate large teacher and has been shown to improve generalization in vision transformers and language models through a born-again network training process.