Knowledge Distillation

The student model is trained to match the teacher's softened probability distribution over classes, not just the ground-truth hard label. A temperature parameter (T) in the final softmax layer controls the softness of these probabilities. Higher temperatures reveal the dark knowledge encoded in the relative probabilities of incorrect classes—for example, a teacher might assign a 10⁻³ probability to a visually similar but incorrect class, teaching the student about inter-class similarities that one-hot labels completely miss.

The training objective combines two loss terms:

• Distillation Loss: Kullback-Leibler divergence between the softened teacher and student output distributions, scaled by T² to maintain gradient magnitudes
• Student Loss: Standard cross-entropy between student outputs and ground-truth labels

The weighted sum L = α · L_soft + (1-α) · L_hard allows practitioners to balance the importance of mimicking the teacher versus learning directly from labels. Typical α values range from 0.7 to 0.9, prioritizing the teacher's richer supervisory signal.

The temperature hyperparameter critically shapes knowledge transfer:

• T = 1: Standard softmax, no softening applied
• T > 1: Produces softer distributions, exposing more of the teacher's dark knowledge about class relationships
• T → ∞: Approaches a uniform distribution, losing discriminative information

Typical values range from T = 2 to T = 20 depending on task complexity. For RF modulation classification with many similar constellation patterns, higher temperatures (T = 8-15) help the student learn subtle inter-class distinctions that hard labels obscure.

Beyond output-level matching, intermediate representations can be transferred:

• Hint-based training: The student learns to regress the teacher's intermediate feature maps using a guided layer, aligning internal representations before final output distillation
• Attention transfer: Matching spatial attention maps between teacher and student networks forces the student to focus on the same input regions
• Relational distillation: Preserving pairwise distance relationships between samples in the teacher's embedding space, capturing structural knowledge about the data manifold

For RF applications, feature-based distillation helps student models learn the same signal processing hierarchies the teacher discovered.

Two primary operational modes exist:

• Offline distillation: A pre-trained, frozen teacher model transfers knowledge to the student. This is the standard approach and requires storing or accessing the teacher during student training
• Online distillation: Teacher and student are trained simultaneously, with the teacher updating alongside the student. This eliminates the need for a pre-trained teacher but increases training complexity
• Self-distillation: A single model acts as both teacher and student, using its own predictions from previous epochs or deeper layers to regularize training. This has shown particular promise in few-shot RF signal classification where labeled data is scarce

Knowledge distillation achieves significant model compression for on-device RF deployment:

• Parameter reduction: Student models typically contain 5-20x fewer parameters than their teachers while retaining 95-98% of teacher accuracy on modulation classification tasks
• Inference latency: Distilled models for automatic modulation classification on FPGA-based edge devices achieve 3-8x speedup compared to the original teacher
• Energy efficiency: On resource-constrained SDR platforms, distilled student models reduce inference energy consumption by 4-10x, critical for battery-operated spectrum sensing nodes
• Memory footprint: Student models often fit within 50-200 KB of SRAM, enabling deployment on microcontroller-class devices for real-time RF fingerprinting

A training method that simulates low-precision inference during the forward pass, enabling the model to learn parameters robust to quantization error. Unlike post-training methods, QAT maintains accuracy by exposing the network to the effects of INT8 or INT4 rounding during optimization. This is critical for deploying neural receivers on integer-only NPUs and microcontrollers where floating-point units are unavailable.

The systematic removal of redundant or low-magnitude connections in a neural network. Unstructured pruning zeroes out individual weights, while structured pruning removes entire channels or filters for direct hardware acceleration. When combined with distillation, a pruned student model can recover accuracy lost during sparsification by learning from the dense teacher's soft outputs.

An automated process that explores a defined search space of network topologies to discover optimal architectures under specific hardware constraints. Hardware-aware NAS incorporates latency and energy feedback directly from the target accelerator. The discovered compact architecture often serves as the student model in a subsequent distillation step to further boost accuracy.

A three-stage optimization framework that sequentially applies pruning, trained quantization, and Huffman coding to achieve state-of-the-art compression rates. Knowledge distillation can be integrated at any stage to transfer dark knowledge from the original uncompressed model to the progressively smaller variants, preventing cumulative accuracy degradation.

Mathematical techniques like low-rank factorization that approximate high-dimensional weight tensors with smaller constituent factors. By decomposing a large convolutional kernel into a sequence of smaller operations, parameter count is drastically reduced. Distillation provides a complementary signal, teaching the factorized student to mimic the original tensor's functional behavior rather than just its structural approximation.

The empirical finding that dense, randomly-initialized networks contain sparse subnetworks—winning tickets—that can train to comparable accuracy in isolation. These subnetworks represent a form of implicit architecture discovery. A winning ticket can serve as a highly efficient student initialization, with distillation used to accelerate convergence and improve final performance.