Model Compression via Distillation

The foundational distillation method where the student model is trained to mimic the teacher model's raw, pre-softmax output logits. A temperature parameter (T) is used to soften the teacher's output distribution, creating soft targets that reveal inter-class relationships (e.g., 'cat' is more similar to 'dog' than to 'airplane'). The student minimizes a Kullback-Leibler Divergence Loss between its softened outputs and the teacher's. This is the core technique introduced in the seminal 2015 paper 'Distilling the Knowledge in a Neural Network' by Hinton et al.

This class of techniques moves beyond final outputs to transfer knowledge from the teacher's internal representations.

• Feature Mimicking / Hint Learning: The student is trained to replicate the feature activations from specific intermediate hint layers of the teacher. This often requires an adaptation layer (e.g., a 1x1 convolution) to match dimensionalities.
• Attention Transfer: Forces the student to produce similar attention maps as the teacher. For vision models, this means spatial attention from convolutional layers; for transformers, it involves aligning the patterns in multi-head attention matrices. This transfers the teacher's 'focus' or saliency patterns.

These paradigms relax the requirement for a static, pre-trained teacher.

• Online Distillation: The teacher and student are co-trained simultaneously in a single end-to-end process. The teacher's weights are updated via a standard task loss, while the student learns from the evolving teacher. This is more efficient than the traditional two-stage process.
• Self-Distillation: A model distills knowledge from itself. In Born-Again Networks, an identical-architecture student is trained from the predictions of its teacher predecessor, often achieving superior performance. Other variants involve distilling knowledge from deeper layers to shallower layers within the same network.

Advanced techniques for complex or constrained scenarios.

• Multi-Teacher Distillation: A single student learns from an ensemble of teachers, each potentially specialized in different tasks or data domains. The student must amalgamate this diverse knowledge, often by averaging soft targets or using weighted losses.
• Data-Free Distillation: The student is trained without access to the original training data. A generator network creates synthetic samples designed to maximize the teacher's response variance or mimic its training data distribution. This is critical for privacy-sensitive applications or when data is proprietary.

Methods that use relational or adversarial objectives for richer knowledge transfer.

• Contrastive Representation Distillation: Encourages the student's feature representations to have a similar relational structure as the teacher's. Positive pairs (e.g., different augmentations of the same image) should have similar embeddings in both models, while negative pairs should be dissimilar. This transfers structural knowledge of the feature space.
• Adversarial Distillation: Employs a Generative Adversarial Network (GAN) framework. A discriminator is trained to distinguish between feature representations from the teacher vs. the student. The student is trained to generate features that 'fool' the discriminator, thereby matching the teacher's feature distribution.

Specialized distillation methods for particular model architectures and deployment goals.

• Sequence-Level Distillation: Used for sequence-to-sequence models (e.g., machine translation). The student learns to match the teacher's distribution over entire output sequences, often using beam search outputs from the teacher as soft targets.
• Quantization-Aware Distillation (QAT-D): A joint optimization where distillation is performed while simulating the effects of low-precision quantization (e.g., INT8). The student learns a robust, quantization-friendly representation from the teacher, directly optimizing for efficient on-device inference.
• Cross-Modal Distillation: Transfers knowledge from a teacher trained on one data modality (e.g., RGB images) to a student using a different, often cheaper, modality (e.g., infrared or grayscale).