Knowledge Distillation

The foundational two-model paradigm where a large, pre-trained teacher network generates soft probability distributions over outputs. The smaller student model is trained not just on hard labels but on these soft targets, which encode rich inter-class similarities learned by the teacher. The student minimizes a composite loss function combining the standard cross-entropy loss with a distillation loss, typically Kullback-Leibler divergence, weighted by a temperature hyperparameter.

The most common form of knowledge transfer, where the student mimics the final output layer of the teacher. This approach is straightforward to implement but loses intermediate representational information. It is particularly effective for classification tasks where the relative probabilities of incorrect classes carry significant semantic meaning about the input data manifold.

Transfers knowledge by aligning the intermediate feature representations of the student and teacher networks. A regression loss, such as L2 distance, is applied between the teacher's hidden layer activations and a learned linear projection of the student's corresponding layer. This method captures the hierarchical feature extraction process, enabling the student to learn better internal representations beyond just mimicking final outputs.

Preserves the structural relationships between data samples rather than individual outputs. The student learns to replicate the similarity matrices generated by the teacher, capturing how the teacher organizes the embedding space. Techniques include:

• Instance Relationship Graph: Matching pairwise distances between samples in a batch.
• Flow of Solution Procedure (FSP): Matching Gram matrices between two layers to preserve the problem-solving process.

A peer-learning paradigm where the teacher and student models are trained simultaneously from scratch, rather than sequentially. In architectures like Deep Mutual Learning, a cohort of untrained student networks learn collaboratively by matching each other's soft predictions. This eliminates the need for a pre-trained, computationally expensive teacher and often yields students that outperform individually trained counterparts.