Knowledge Distillation

The most straightforward approach where the student model learns to mimic the soft labels (output probability distributions) of the teacher.

• Soft Targets: Instead of training on hard one-hot labels, the student uses the teacher's softened logits, which contain rich information about inter-class similarities.
• Temperature Scaling: A hyperparameter T is applied to the softmax function to smooth the distribution, revealing dark knowledge about which classes the teacher considers similar.
• Loss Function: Typically combines a distillation loss (KL divergence between student and teacher outputs) with a standard cross-entropy loss against ground truth.
• Use Case: Effective for classification tasks where the relative probabilities between incorrect classes carry valuable structural information.

Trains the student to replicate the intermediate representations and feature maps of the teacher, not just the final output.

• Hint Layers: Specific intermediate layers in the teacher are selected as targets for the student to match, providing richer supervision than output alone.
• Attention Transfer: The student learns to mimic the teacher's attention maps, forcing it to focus on the same salient regions of the input.
• Dimensionality Mismatch: A learned regressor or projection layer is often needed to align the student's smaller feature dimensions with the teacher's larger ones.
• Benefit: Captures the internal reasoning process of the teacher, often leading to better generalization than response-based methods alone.

Preserves the structural relationships between data samples as learned by the teacher, rather than individual outputs.

• Instance Relationship Graphs: The student learns to maintain the same relative distances and angles between sample embeddings as the teacher produces.
• Flow of Solution Process (FSP): Captures the relationship between feature maps of two layers by computing the Gram matrix, enforcing the student to learn the same problem-solving flow.
• Similarity-Preserving: Directly constrains the similarity matrices produced by batches of data to be identical between teacher and student.
• Advantage: Focuses on the geometry of the learned representation space, which is often more invariant and transferable than absolute feature values.

A collaborative paradigm where the teacher and student models are trained simultaneously in a single end-to-end process, rather than sequentially.

• Deep Mutual Learning: Multiple peer networks train together, each acting as both teacher and student to the others, improving all models beyond isolated training.
• Co-Distillation: A large model and a small model train concurrently, with the large model distilling knowledge to the small one in real-time during the training loop.
• Architecture: Eliminates the need for a pre-trained, frozen teacher, reducing the overall training pipeline complexity and storage costs.
• Efficiency: Particularly useful when no suitable pre-trained teacher exists for a highly specialized domain.

A technique where a model distills knowledge from its own deeper layers or earlier training epochs to improve itself without an external teacher.

• Born-Again Networks: A model with identical architecture is trained sequentially, where each generation is distilled from the previous one, yielding performance gains with no increase in inference cost.
• Deep Supervision: Early exit branches are added to a network and trained to mimic the final output, regularizing the lower layers and improving gradient flow.
• Label Smoothing Connection: Can be interpreted as a form of self-distillation where the model learns from its own softened predictions.
• Simplicity: Requires no separate teacher model, simplifying the deployment pipeline while still providing meaningful accuracy improvements.

Compresses a teacher model into a student when the original training data is unavailable due to privacy, security, or bandwidth constraints.

• Synthetic Sample Generation: A generator network creates artificial inputs that maximize the teacher's response discrepancy, providing training material for the student.
• Model Inversion: Reconstructs representative inputs by optimizing random noise to activate specific neurons or output classes in the teacher.
• Adversarial Training: A generative adversarial network is used where the generator produces samples that the teacher is confident about, and the student learns to match.
• Critical Application: Essential for deploying compact models in regulated industries where raw data cannot leave a secure enclave.