Knowledge Distillation

The foundational and most common form of knowledge distillation. The student model is trained to mimic the final output probabilities (logits) of the teacher model, not just the hard class labels.

• Soft Targets: Instead of a one-hot encoded label (e.g., [0, 0, 1, 0]), the student learns from the teacher's softened probability distribution (e.g., [0.01, 0.04, 0.90, 0.05]) using a high temperature parameter in the softmax function.
• Loss Function: Combines a standard cross-entropy loss with the ground truth (hard labels) and a distillation loss (e.g., Kullback-Leibler divergence) that measures the difference between the student's and teacher's softened outputs.
• Primary Goal: Teaches the student the relative relationships between classes as perceived by the more nuanced teacher.

Also known as hint-based training or intermediate representation distillation. This technique transfers knowledge from the teacher's internal feature maps or activations, not just its final predictions.

• Mechanism: A projection layer (or 'hint' and 'guide' layers) is added to align the dimensions of the student's and teacher's intermediate feature tensors. The student is trained to minimize the difference (e.g., using Mean Squared Error) between its transformed features and the teacher's.
• Rationale: The teacher's internal representations contain rich, structured information about the data that is lost by the final classification layer. Mimicking these features can lead to faster convergence and better generalization in the student.
• Common Use: Particularly effective for compressing convolutional neural networks (CNNs) in computer vision, where features from middle layers are highly informative.

A specialized technique for Transformer-based models (e.g., BERT, GPT). It transfers the attention patterns learned by the teacher model, which encode how the model focuses on different parts of the input sequence.

• Process: The student's multi-head attention matrices are trained to match the teacher's corresponding attention matrices. This can be done by minimizing the Mean Squared Error or a divergence measure between the attention distributions.
• Benefit: Attention maps capture linguistic relationships and dependencies. By mimicking them, the student learns a similar structural understanding of language, which is crucial for tasks like translation, summarization, and question answering.
• Variants: Can be applied to both self-attention and cross-attention mechanisms in encoder-decoder architectures.

A variant where the teacher and student are the same model architecture, or where a model distills knowledge into itself. This paradox can improve a model's own performance or create a more efficient version.

• Same-Architecture Self-Distillation: A model is trained normally, then used as a teacher to train another instance of the same architecture from scratch. This iterative process can often boost the final student's accuracy.
• Progressive Self-Distillation: A single model is trained, and its knowledge at earlier training checkpoints is used to guide its own later training via a distillation loss, acting as a form of regularization.
• Primary Advantage: Does not require a separate, larger teacher model. It leverages the idea that a model's own predictions contain richer information than one-hot labels.

A dynamic, single-stage training process where the teacher and student models are co-trained simultaneously, rather than using a fixed, pre-trained teacher.

• Mechanism: An ensemble of models (the 'peers') are trained in parallel. The knowledge distilled into any one student model is an aggregate of the outputs from all other models in the ensemble at that training step.
• Benefits:
  • Eliminates the need for a costly, two-stage training pipeline.
  • The teaching signal evolves and improves alongside the student models.
  • Can be more efficient and lead to better final performance than offline distillation with a static teacher.
• Common Framework: Deep Mutual Learning is a prominent example, where a cohort of models teach each other throughout training.

An advanced technique where the student model is trained without access to the original training data, using only the pre-trained teacher model. This is critical for privacy-sensitive or proprietary data scenarios.

• Core Challenge: Generating synthetic data that elicits a useful teaching signal from the teacher.
• Methods:
  • Generator-Based: A generative model (e.g., a GAN) is trained to produce inputs that maximize the disagreement or information gain from the teacher, which are then used to train the student.
  • Optimization-Based: Synthetic data samples are directly optimized via gradient ascent to match statistics (e.g., batch normalization layer statistics) of the teacher's activations recorded during its original training.
• Use Case: Essential for compressing models where the training data is no longer available or cannot be shared due to regulations like GDPR.

Self-distillation is a variant where the teacher and student models are identical in architecture. The model is trained to match its own predictions from a previous training iteration or a differently perturbed version of itself. This can act as a form of regularization and model refinement.

• Iterative Self-Distillation: A model trained in one generation becomes the teacher for the next generation of the same architecture.
• Benefits: Can improve the model's calibration and generalization without requiring a larger, external teacher. It simplifies the distillation pipeline by removing the need for two distinct models.