Knowledge Distillation

The foundational setup where a large, pre-trained teacher model generates soft probability distributions over outputs. A smaller student model is trained not just on the hard labels but on these soft targets, which capture the teacher's nuanced decision boundaries and inter-class similarities. This transfers dark knowledge that is not present in the one-hot ground truth.

The most common form of distillation, focusing solely on the final output layer. The student minimizes the divergence between its predicted probability distribution and the teacher's distribution. A temperature parameter (T) in the softmax function softens the probabilities, revealing the relative confidence the teacher assigns to incorrect classes. Key loss components include:

• Kullback-Leibler divergence between softened student and teacher outputs
• Cross-entropy loss against the ground truth hard labels

Instead of matching only final outputs, the student learns to mimic the intermediate feature representations of the teacher. A regression loss minimizes the distance between the student's hidden layer activations and the teacher's corresponding layers. This is critical when the student has a different depth or width, requiring a projection layer to map the student's feature space to the teacher's dimensionality.

Transfers the structural relationships learned by the teacher. Instead of matching individual examples, the student learns to preserve the mutual relations between data samples in the teacher's embedding space. Techniques include:

• Similarity-preserving loss: Matching the Gram matrices of activations within a batch
• Relational Knowledge Distillation (RKD): Preserving distance-wise and angle-wise relations between triplet samples

A critical application in retrieval systems where a computationally heavy cross-encoder (which processes query-document pairs jointly) acts as the teacher. The student is a bi-encoder that encodes queries and documents independently for fast ANN search. The bi-encoder is trained to mimic the cross-encoder's relevance scores, enabling low-latency semantic search without sacrificing the precision of joint attention mechanisms.

Offline distillation uses a frozen, pre-trained teacher to guide a student in a standard two-phase process. Online distillation trains both models simultaneously in a single phase, where the teacher and student co-evolve. A variant, self-distillation, uses the same architecture for both roles, where a deeper network's predictions guide a shallower instance of itself, often improving generalization without an external teacher.