Knowledge Distillation

The most fundamental distillation technique. The student model is trained to directly mimic the softened output logits (pre-softmax activations) of the teacher model. A temperature parameter T is applied to the teacher's softmax to create a richer probability distribution (soft targets) that reveals dark knowledge—the relative similarities between classes. The standard loss function is Kullback-Leibler (KL) Divergence, which measures the difference between the two probability distributions.

• Primary Use: General model compression and transfer learning.
• Key Component: Temperature scaling (T > 1) to soften teacher outputs.
• Example: Training a compact mobile model to match the logits of a massive ensemble model.

Also known as hint-based or representation distillation. Instead of just the final outputs, the student model is guided to replicate the intermediate feature maps or activations from specific layers of the teacher model. This transfers the teacher's internal representational knowledge.

• Mechanism: Aligns the student's feature space to the teacher's using a regression loss (e.g., Mean Squared Error) on selected layer outputs.
• Advantage: Provides a stronger, more direct learning signal than logits alone, often leading to better student performance.
• Seminal Work: FitNets introduced 'hint training,' where a student's guided layer learns to match a teacher's intermediate 'hint' layer.

A specialized form of feature distillation highly effective for transformer-based architectures (e.g., BERT, ViT). The student model is trained to mimic the attention maps or patterns of the teacher model. This transfers the teacher's contextual understanding and focus mechanisms.

• What are Attention Maps?: Matrices that indicate how much focus the model places on different parts of the input when generating a representation for a specific token or pixel.
• Process: A distillation loss is applied between the attention matrices of corresponding teacher and student layers.
• Example: TinyBERT uses attention distillation as a core component of its two-stage compression framework for BERT.

A paradigm where a model serves as both its own teacher and student. Knowledge is distilled within the same model or between identical models at different training stages.

• Common Variants:
  • Depth-based: Distilling knowledge from deeper layers to shallower layers of the same network.
  • Iterative: Using a model checkpoint from a later training epoch to teach the same model earlier in training.
  • Uniform: Training multiple identical models in parallel and distilling knowledge between them.
• Advantage: Does not require a separate, larger teacher model. Can regularize training and improve a model's own generalization.

A dynamic variant where the teacher and student models are trained simultaneously, co-evolving together, rather than using a static, pre-trained teacher.

• Contrast with Standard Distillation: In standard (offline) distillation, the teacher is fixed. In online distillation, the teacher's weights are updated during training, providing a continuously improving supervisory signal.
• Architecture: Often implemented with an ensemble of peer models where each model learns from the ensemble's average predictions.
• Benefit: More flexible and can lead to better final performance, as the teacher adapts alongside the student.

The application of knowledge distillation as a regularization technique in continual learning to mitigate catastrophic forgetting. When learning a new task, the model's predictions on new data are compared to its own predictions on that data before learning the new task.

• Core Algorithm: Learning without Forgetting (LwF). The model's outputs on new task data, using its old parameters, are saved as soft targets. A distillation loss between these old targets and the new predictions is added to the new task's classification loss.
• Function: This loss penalizes the model for changing its behavior on new data in a way that would disrupt its knowledge of old tasks, effectively 'anchoring' previous knowledge.
• Key Feature: Enables rehearsal of past knowledge without storing raw data, only requiring a forward pass to generate soft labels.

A core application in continual learning systems. When a model learns a new task, its outputs on previous tasks are used as soft targets to regularize training, preserving old knowledge.

• Mechanism: The model being updated serves as both the student (learning the new task) and the teacher (providing targets for old tasks), a process called self-distillation.
• Algorithm: Learning without Forgetting (LwF) is the seminal algorithm for this, using a knowledge distillation loss on the model's own predictions for old task data to prevent performance degradation.

Distilling the collective knowledge of a computationally expensive ensemble of models (multiple teachers) into a single, manageable student model. The student learns from the averaged or combined softened predictions of the ensemble.

• Key Benefit: Captures the robustness and improved accuracy of an ensemble without the multiplicative inference cost.
• Process: The ensemble's predictions are aggregated (e.g., averaged logits) to form a richer, more reliable set of soft targets for the student than any single model could provide.

Transferring learned representations from a model with one architecture (e.g., a CNN) to a student with a different, often more efficient architecture (e.g., a MobileNet). This goes beyond simple compression to architectural innovation.

• Feature Distillation: Techniques like FitNets and Attention Transfer are used here, where the student mimics the teacher's intermediate feature maps or attention patterns.
• Use Case: Transferring knowledge from a large, state-of-the-art research model to a novel, hardware-optimized student architecture designed for production.

Using distillation to create a central model without direct access to raw, sensitive training data. Local models (teachers) are trained on private data silos, and only their soft predictions on a public, unlabeled dataset are shared to train a central student.

• Privacy Benefit: The raw private data never leaves its local environment; only non-sensitive model outputs are transmitted.
• Federated Context: This approach can reduce communication costs and privacy risks compared to sharing model weight updates in traditional federated learning.

Using a powerful teacher model to generate higher-quality soft labels for noisy or weakly-labeled datasets, which are then used to train a student. The teacher acts as a smart annotator.

• Process: The teacher's soft targets on unlabeled or noisy data capture class similarities and confidence, providing a better learning signal than raw, potentially incorrect one-hot labels.
• Application: Improving model training in domains with inherently noisy labels (e.g., web-scraped data, user-generated content) or for semi-supervised learning where a small labeled set is used to label a larger unlabeled set via distillation.

Catastrophic forgetting is the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new data. This is the core problem continual learning systems aim to solve.

• Mechanism: New gradient updates overwrite weights critical for old tasks.
• Analogy: Like learning Spanish and completely forgetting how to speak French.
• Mitigation: Addressed by techniques like knowledge distillation, elastic weight consolidation, and experience replay.

Experience replay is a continual learning mechanism where a subset of past training data is stored in a memory buffer and replayed during new learning to consolidate old knowledge.

• Core Function: Interleaves old and new examples during training.
• Memory Buffer: A fixed-size storage (e.g., 1% of total data) using strategies like reservoir sampling.
• Relation to Distillation: Often combined with distillation losses, where the model's own past predictions on replayed data serve as soft targets.

Elastic Weight Consolidation is a regularization-based continual learning algorithm that estimates the importance (Fisher information) of each network parameter to previous tasks and penalizes changes to important weights.

• Key Idea: Some weights are "elastic" (less important) and can change, while others are "plastic" (critical) and should be consolidated.
• Difference from Distillation: EWC operates directly on the parameter space, while distillation operates on the output or feature space.
• Use Case: Effective for task-incremental learning where task boundaries are known.

Learning without Forgetting is a seminal algorithm that uses knowledge distillation as its core mechanism for continual learning. The current model's outputs on new data for old tasks are used as soft targets to preserve previous knowledge.

• Process: 1) Record model's pre-softmax logits for new data on old task heads. 2) Learn new task. 3) Use a distillation loss to match old outputs.
• Advantage: Does not require storing raw past data (exemplar-free).
• Limitation: Performance depends on data overlap between old and new tasks.

Generative replay is a rehearsal technique where a generative model (e.g., GAN, VAE) is trained to produce synthetic data representative of past tasks. This pseudo-data is then replayed to the main model.

• Core Concept: The generator acts as a pseudo-memory of past data distributions.
• Pseudo-Rehearsal: A related method where the main model itself generates its own pseudo-data for replay.
• System Design: Requires training and maintaining a separate generative model in parallel with the main classifier.

The plasticity-stability trade-off is the fundamental challenge in designing continual learning systems. A model must balance two competing objectives:

• Plasticity: The ability to rapidly learn new information and adapt to new tasks.
• Stability: The ability to retain previously acquired knowledge and resist catastrophic forgetting.

All continual learning algorithms, including knowledge distillation, are essentially strategies for managing this trade-off. Over-regularizing for stability leads to intransigence (inability to learn new things), while excessive plasticity causes forgetting.