Knowledge Distillation for Edge

The teacher is typically a vast, over-parameterized ensemble or a massive transformer, while the student is a lightweight architecture like MobileNetV3, EfficientNet, or a pruned DistilBERT. The distillation process bridges the capacity gap by transferring dark knowledge—the relative probabilities of incorrect classes in the teacher's softmax output—which encodes a rich similarity structure over the data that the student cannot learn from hard labels alone.

A critical hyperparameter, the temperature (T), is introduced into the softmax function of the teacher. A T > 1 softens the probability distribution, revealing the inter-class relationships learned by the teacher.

• High T: Produces a softer distribution, exposing more of the teacher's dark knowledge but also more noise.
• Low T: Approaches a hard label distribution, reducing the information transferred. The student is trained with the same high temperature, but evaluated with T=1.

The student model is optimized using a weighted combination of two distinct loss terms:

• Distillation Loss: The Kullback-Leibler (KL) divergence between the softened outputs of the teacher and student. This transfers the teacher's generalization capabilities.
• Student Loss: The standard cross-entropy loss between the student's hard predictions (T=1) and the ground-truth labels. This anchors the student to the task objective. The balance is controlled by a hyperparameter α (alpha).

Beyond matching final output probabilities, the student can be trained to mimic the teacher's intermediate feature representations. A loss function minimizes the distance (e.g., L2 norm or Maximum Mean Discrepancy) between the student's and teacher's feature maps at specific bottleneck layers. This is particularly effective for edge models that must replicate the hierarchical feature extraction of a deeper network, often using a learned adaptation layer to project the student's narrower feature space to the teacher's dimensionality.

Edge deployment often occurs in environments where the original, privacy-sensitive training data is inaccessible. Data-free knowledge distillation synthesizes surrogate training samples by inverting the teacher model's learned parameters. The student is trained on these generated inputs, which are optimized to maximize the teacher's response, effectively extracting its knowledge without any real data. This is crucial for deploying models in regulated industries like healthcare and finance.

To bridge the gap between a full-precision (FP32) teacher and an INT8-quantized student, quantization-aware distillation integrates simulated quantization operations directly into the student's training graph. The student learns to be robust to the quantization error it will encounter on a Neural Processing Unit (NPU) or DSP. This co-optimization yields significantly higher post-quantization accuracy than training a full-precision student and applying post-training quantization as a separate step.

The foundational two-model training paradigm where a large, high-capacity teacher model generates soft targets for a compact student model. The student is trained not on hard one-hot labels, but on the teacher's output probability distribution over all classes. This soft label distribution encodes rich inter-class similarity knowledge—for example, that a cat is more similar to a dog than to a truck—which is lost in hard labels. The student minimizes a composite loss function combining the standard cross-entropy loss with a distillation loss (often Kullback-Leibler divergence) against the teacher's softened logits.

A critical hyperparameter T applied to the softmax function that controls the softness of the teacher's output probability distribution. Higher temperatures (T > 1) produce softer, more uniform distributions that expose the dark knowledge of inter-class relationships. The softmax with temperature is computed as: p_i = exp(z_i / T) / Σ exp(z_j / T). During distillation, the teacher and student logits are divided by the same temperature before computing the KL divergence loss. At inference, the student uses T=1. Typical values range from 2 to 20, tuned based on the task complexity.

The implicit, non-obvious information encoded in a teacher model's soft output distribution that goes beyond the ground-truth hard label. This includes: class similarity structures (which classes are easily confused), generalization patterns learned from the training data, and feature hierarchies that the teacher has internalized. Dark knowledge is what makes distillation effective—it provides significantly more bits of information per training example than a one-hot label. For a 1000-class problem, a hard label provides ~10 bits, while a full soft distribution provides ~1000 real-valued targets.

The student model is optimized using a weighted combination of two losses:

• Distillation Loss: KL divergence between the softened teacher and student output distributions, scaled by T² to maintain gradient magnitudes
• Student Loss: Standard cross-entropy between student predictions and ground-truth hard labels

The balance is controlled by a hyperparameter α (alpha), typically set between 0.5 and 0.9, weighting the distillation loss more heavily. The T² scaling factor is mathematically necessary because the gradient of the distillation loss scales as 1/T² when using temperature scaling.

An extension beyond output logit matching where the student is trained to mimic the teacher's intermediate feature representations. Common approaches include:

• Hint-based training: The student's middle layer activations are regressed to match the teacher's corresponding layer using an L2 loss
• Attention transfer: The student learns to replicate the teacher's spatial attention maps
• Relation-based distillation: Preserving the mutual relationships between samples in the teacher's embedding space This is particularly effective for edge deployment because it transfers structural knowledge about how the model processes inputs, not just final predictions.

A variant where the teacher and student share the same architecture, and the model is iteratively trained using its own previous predictions as soft targets. The process works as follows:

• Train a model to convergence on hard labels (generation 1)
• Retrain the same architecture from scratch using generation 1's soft predictions as targets (generation 2)
• Repeat for multiple generations Surprisingly, each generation often outperforms its predecessor despite identical capacity. This works because the soft targets provide a richer, denoised training signal. Self-distillation is particularly useful for edge scenarios where no large teacher model is available.