Knowledge Distillation

The framework involves two distinct models: a large, pre-trained teacher and a compact, untrained student. The student is optimized using a composite loss function that combines the standard ground-truth loss with a distillation loss, which penalizes the divergence between the student's softened output distribution and the teacher's. Crucially, the teacher's weights are frozen during this process.

Instead of training only on hard labels, the student learns from soft targets—the teacher's output probability vector. A temperature hyperparameter (T) is applied to the softmax function to soften these probabilities, revealing the relative similarities between incorrect classes. This 'dark knowledge' provides richer information per sample, teaching the student how the teacher generalizes rather than just memorizing final predictions.

The student is optimized by minimizing the discrepancy between its softened logits and the teacher's. Common loss functions include:

• Kullback-Leibler (KL) Divergence: The standard choice for matching softened probability distributions.
• Mean Squared Error (MSE): Applied directly to the logits before the softmax layer, preserving the teacher's internal representation geometry.
• Cosine Embedding Loss: Used to align the direction of feature vectors in the penultimate layer.

Offline distillation is the traditional two-phase process where a large teacher is pre-trained first, and then its knowledge is transferred to a student. Online distillation updates both models simultaneously in a single end-to-end training loop, often using a peer-ensemble strategy where multiple students learn from each other. Self-distillation is a variant where a model uses its own earlier, deeper checkpoints as the teacher.

Beyond matching final output probabilities, the student can be trained to replicate the teacher's intermediate feature representations. This involves minimizing the distance between activation maps of corresponding layers. Techniques include:

• Attention Transfer: Forcing the student to mimic the teacher's spatial attention maps.
• Hint-Based Learning: Using a regressor to map the student's narrower hidden layer to the teacher's wider dimension before computing the loss.

Knowledge distillation is a critical enabler for TinyML and edge AI. It allows a cumbersome cloud-scale model (e.g., a massive transformer) to train a highly efficient model suitable for microcontrollers or mobile GPUs. The student architecture is often a hand-crafted compact CNN or a MobileNet, achieving significant reductions in inference latency and memory footprint without catastrophic accuracy loss compared to direct training on limited data.

The foundational framework where a large, high-capacity teacher model generates soft labels—probability distributions over classes rather than hard one-hot labels—that encode inter-class relationships. The compact student model is trained to match these soft targets, capturing the teacher's generalization ability. Key mechanisms include:

• Softmax temperature scaling: A temperature parameter T > 1 softens the output distribution, revealing dark knowledge about class similarities
• Loss function combination: Typically combines distillation loss (KL divergence between student and teacher softmax outputs) with standard cross-entropy on ground truth
• Intermediate layer distillation: Aligning feature maps or attention patterns at multiple network depths, not just final outputs

A compression technique that reduces the numerical precision of weights and activations from 32-bit floating point to lower bit-widths such as INT8 or FP16. This directly decreases memory footprint and enables faster integer arithmetic on edge accelerators. Common approaches:

• Post-Training Quantization (PTQ): Applied after training with a small calibration dataset to determine optimal scaling factors
• Quantization-Aware Training (QAT): Simulates quantization noise during training, allowing the model to adapt and recover accuracy
• Dynamic range quantization: Weights are statically quantized while activations are quantized on-the-fly during inference
• Typical memory reduction: 4x when moving from FP32 to INT8, with latency improvements of 2-4x on supported hardware

An optimization strategy that identifies and removes redundant or low-magnitude parameters from a neural network. Pruning creates sparse weight matrices that reduce both storage and computation. Major categories include:

• Unstructured pruning: Zeroing out individual weights based on magnitude thresholds, producing irregular sparsity patterns
• Structured pruning: Removing entire channels, filters, or attention heads, which directly reduces tensor dimensions and is hardware-friendly
• Iterative magnitude pruning: Alternating between pruning and fine-tuning cycles to gradually increase sparsity while recovering accuracy
• Lottery Ticket Hypothesis: Suggests that dense networks contain sparse subnetworks that can be trained in isolation to match original performance
• Typical compression: 50-90% parameter reduction with minimal accuracy loss when combined with fine-tuning

An automated methodology that searches for optimal model architectures while incorporating deployment constraints directly into the optimization loop. Unlike traditional NAS that maximizes accuracy alone, hardware-aware NAS jointly optimizes for:

• Inference latency on target silicon (e.g., measured in milliseconds on a specific FPGA or microcontroller)
• Energy consumption per inference, critical for battery-powered edge devices
• Memory footprint including peak activation memory and persistent weight storage
• Operator compatibility: Ensuring all operations in the discovered architecture are supported by the target inference runtime (TensorRT, TFLite Micro, etc.)
• Search spaces typically include kernel sizes, channel counts, layer depths, and skip connection patterns

A graph-level optimization that combines multiple consecutive neural network operations into a single computational kernel. This eliminates intermediate memory reads and writes, significantly reducing memory bandwidth bottlenecks on edge devices. Common fusion patterns:

• Conv-BatchNorm-ReLU: Fusing convolution, batch normalization, and activation into one kernel
• MatMul-Bias-Add: Combining matrix multiplication with bias addition and residual connections
• Attention head concatenation: Merging multi-head attention computations in transformer architectures
• Element-wise operation chains: Collapsing sequences of add, multiply, and activation functions
• Typical latency reduction: 15-30% on GPU and FPGA deployments, with even greater gains on memory-constrained microcontrollers