Model Distillation

The most straightforward approach where the student model is trained to mimic the final output distribution of the teacher.

• Soft Targets: Instead of training on hard one-hot labels, the student learns from the teacher's softmax output probabilities, which contain rich inter-class similarity information.
• Softmax Temperature: A hyperparameter T is introduced to soften the probability distribution, revealing the dark knowledge of non-target classes. Higher T produces a softer distribution.
• Loss Function: Typically combines a distillation loss (KL divergence between student and teacher softmax outputs) with a standard student loss against ground-truth hard labels.
• Example: In image classification, a ResNet-50 student learns not just that an image is a 'cat', but that the teacher assigned small probabilities to 'dog' and 'fox', capturing visual similarity.

Trains the student to replicate the internal feature representations of the teacher, not just the final output.

• Intermediate Layers: The student is penalized for differences between its hidden layer activations and those of the teacher's corresponding layers.
• Hint Layers: A selected intermediate layer from the teacher used as a supervised signal for the student's guided layer.
• Dimensionality Mismatch: A regressor layer is often used to map the student's feature dimensions to match the teacher's hint layer dimensions.
• Benefit: Provides richer supervision than response-based methods alone, especially for very deep teacher networks where the output signal may be too abstract.

Focuses on preserving the mutual relationships and structural dependencies learned by the teacher across data samples or feature maps.

• Instance Relationship Graph: The student learns to preserve the similarity structure between instances in a batch, ensuring that pairs of inputs that produce similar teacher embeddings also produce similar student embeddings.
• Flow of Solution Process (FSP): Captures the relationship between two layers by computing the Gram matrix of their feature maps, then trains the student to match these inter-layer relationships.
• Singular Value Decomposition (SVD) Transfer: Transfers the low-rank structure of teacher weight matrices to the student.
• Key Insight: Instead of asking 'what did the teacher output?', it asks 'how did the teacher structure its knowledge space?'

A collaborative paradigm where the teacher and student models are trained simultaneously, with the teacher model itself being updated during the distillation process.

• Co-Distillation: Multiple models train in parallel, each serving as both a teacher and a student to the others, sharing knowledge bidirectionally.
• Peer Learning: No single model is designated as the fixed teacher; instead, a cohort of untrained models learns collaboratively from their collective predictions.
• Mutual Learning: An ensemble of students is trained together, with each student's loss augmented by a KL divergence term matching the other students' posterior estimates.
• Advantage: Eliminates the need for a pre-trained, static teacher model, reducing the total training pipeline time.

A technique where the teacher and student share the same architecture, using the model's own deeper layers or previous training epochs to supervise its shallower layers or subsequent training iterations.

• Born-Again Networks: A model is trained conventionally, then its own predictions are used as soft targets to train an identical model from scratch, often yielding higher accuracy than the original.
• In-Place Distillation: Uses the deepest classifier of a network to provide supervision signals to earlier, shallower classifiers within the same forward pass.
• Snapshot Distillation: Leverages model snapshots from different points in a single training run (captured via cyclic learning rates) to create an ensemble, then distills the ensemble's knowledge back into the final model.
• Key Benefit: Improves model generalization without requiring a separate, larger teacher architecture.

Performs knowledge transfer when the original training dataset is unavailable, inaccessible due to privacy constraints, or too large to transmit.

• Synthetic Sample Generation: The student model or a separate generator network synthesizes realistic input samples that maximize the teacher's prediction confidence or feature activation statistics.
• Activation Record Matching: The student is trained to match the Batch Normalization statistics (mean and variance) stored in the teacher's layers, without needing any real images.
• Adversarial Generation: A GAN-like framework where a generator creates inputs that cause the teacher to produce a specific output distribution, and the student learns from these synthetic input-output pairs.
• Use Case: Critical for distilling models trained on proprietary medical or financial data where sharing the dataset is legally prohibited.

A compression technique that reduces the numerical precision of a neural network's weights and activations (e.g., from 16-bit to 4-bit) to decrease memory footprint and accelerate inference.

• Post-Training Quantization (PTQ): Applied after training without retraining.
• Quantization-Aware Training (QAT): Simulates quantization during training for higher accuracy.
• Often combined with distillation to recover accuracy lost during precision reduction.

A parameter-efficient fine-tuning method that injects trainable low-rank matrices into a frozen pre-trained model. Unlike distillation, LoRA adapts the original model rather than creating a smaller copy.

• Trains only a fraction of parameters (often <1%).
• Produces lightweight adapters that can be swapped for different tasks.
• QLoRA extends this by backpropagating through a 4-bit quantized model.

A model compression method that removes redundant or low-magnitude weights or entire structures (attention heads, layers) from a neural network.

• Unstructured pruning: Zeroes out individual weights, creating sparse matrices.
• Structured pruning: Removes entire neurons or attention heads for hardware-friendly speedups.
• Often used as a precursor step before distillation to reduce the teacher model's size.

An inference acceleration method that uses a smaller draft model to generate candidate tokens quickly, which are then verified in parallel by a larger target model.

• The draft model is often a distilled version of the target.
• Provides identical output distribution to the large model.
• Reduces latency by 2-3x without quality degradation.

Three distinct compression strategies with different trade-offs:

• Distillation: Trains a compact student to mimic a teacher's behavior. Preserves architectural flexibility.
• Pruning: Removes redundant parts of an existing model. Best for sparsity-aware hardware.
• Quantization: Reduces numerical precision. Often the easiest to apply post-training.
• These techniques are complementary and frequently combined in production pipelines.