Knowledge Distillation

The core framework involves a two-model setup. A high-capacity teacher model (often an ensemble or large foundation model) generates soft labels—probability distributions over classes that capture inter-class similarities. A lightweight student model is then trained to mimic these soft outputs rather than hard ground-truth labels. This transfers the teacher's inductive biases and dark knowledge to the student, enabling the smaller model to generalize far better than it could from raw data alone.

The key mechanism uses a temperature parameter (T) in the final softmax layer to soften the teacher's output distribution. High temperatures produce richer, darker knowledge by revealing the relative similarities between incorrect classes. The student is trained with a composite loss function combining:

• Distillation loss: KL divergence between student and teacher soft targets
• Student loss: Standard cross-entropy with hard labels This ensures the student learns both the correct answer and the teacher's generalization patterns.

In federated learning, clients often have heterogeneous model architectures that cannot be averaged directly. Federated distillation solves this by exchanging model outputs on a public or synthetically generated dataset rather than sharing weights. Each client trains a local student on the ensemble of peer predictions, enabling collaborative learning across non-identical model architectures. This is particularly valuable in cross-device settings where edge hardware constraints vary widely.

When the original training data is unavailable or privacy-restricted, data-free knowledge distillation generates synthetic input samples that maximize the teacher's output response. The student is then trained exclusively on these generated samples and the teacher's corresponding soft labels. This technique is critical in federated scenarios where raw data never leaves local devices, yet the global model must be compressed for efficient on-device deployment.

Knowledge distillation provides an implicit privacy barrier. The student model never accesses raw training data—only the teacher's aggregated output distributions. When combined with differential privacy during teacher training, distillation can produce a student model with formal privacy guarantees. This Private Aggregation of Teacher Ensembles (PATE) framework uses a noisy voting mechanism among multiple teachers to generate privacy-preserving labels for public student training.

Modern variants extend the classic offline paradigm:

• Online distillation: Teacher and student are trained simultaneously, with the teacher co-adapting to the student's learning progress
• Self-distillation: A model distills knowledge from its own deeper layers or previous checkpoints, improving performance without an external teacher
• Mutual distillation: A cohort of peer students learn collaboratively by exchanging soft targets, eliminating the need for a pre-trained teacher entirely

The foundational two-model dynamic where a large, high-capacity teacher model generates soft labels or intermediate representations that a compact student model is trained to mimic. The student learns not just the final hard prediction but the teacher's dark knowledge—the relative probabilities of incorrect classes—which encodes rich similarity structures. This architecture enables deploying models with 10-50x fewer parameters while retaining 95%+ of the original accuracy.

A mechanism where the teacher's output logits are divided by a temperature parameter (T > 1) before softmax, producing a softened probability distribution. Higher temperatures reveal the dark knowledge of inter-class similarities—for example, a car image might show elevated probabilities for 'truck' and 'van' relative to 'apple'. The student is trained to match these soft targets, learning richer representations than from hard one-hot labels alone.

The training objective combines two loss terms: a distillation loss (KL divergence between student and teacher softmax outputs at temperature T) and a student loss (cross-entropy between student predictions and ground-truth hard labels). The weighted sum balances mimicking the teacher against fitting real data. Common variants include: - Feature-based distillation: matching intermediate layer activations - Relation-based distillation: preserving pairwise sample relationships - Adversarial distillation: using a discriminator to align distributions

A counterintuitive technique where a model distills knowledge from itself—using its own predictions from earlier training epochs or deeper layers to regularize and improve its final performance. The model's own softened outputs act as a regularizer, reducing overfitting without requiring a separate teacher. This has been shown to boost accuracy by 1-3% on image classification benchmarks and is particularly effective when combined with progressive resizing or multi-scale training strategies.

A joint optimization approach that combines knowledge distillation with post-training quantization to produce student models optimized for low-precision inference hardware. The teacher guides the student while simulating quantization noise during training, ensuring the compressed model maintains accuracy at INT8 or INT4 precision. This is critical for deploying models on NPUs, mobile devices, and edge hardware where floating-point operations are prohibitively expensive.