Knowledge Distillation

The foundational two-network paradigm where a large, pre-trained teacher model generates soft targets for a compact student model. The teacher is typically a high-capacity ensemble or deep network frozen during distillation. The student, often with 10x-100x fewer parameters, is trained to match the teacher's output distribution rather than hard ground-truth labels. This architecture decouples model capacity during training from inference cost, enabling deployment on FPGAs, embedded ARM processors, and resource-constrained SDR platforms without sacrificing the representational power learned by the teacher.

The key mechanism that transfers dark knowledge from teacher to student. Instead of using hard class labels, the student learns from the teacher's softened probability distribution over all classes. A temperature parameter (T) is applied to the final softmax layer:

• High T (T > 1): Produces softer probabilities, revealing inter-class similarities learned by the teacher
• Low T (T = 1): Approaches standard hard-target training

The soft targets encode rich information about class relationships—for example, that 16-QAM is more similar to 64-QAM than to BPSK—providing significantly more supervisory signal per training example than one-hot labels.

The training objective combines two loss components weighted by a hyperparameter α:

• Soft Loss (KL Divergence): Measures the divergence between the student's softened output and the teacher's softened output, both computed with temperature T
• Hard Loss (Cross-Entropy): Standard supervised loss against ground-truth labels, anchoring the student to correct predictions

The combined loss is:

codeCopy
L = α * L_soft + (1-α) * L_hard

The soft loss gradient must be scaled by T² to maintain proper gradient magnitudes when using temperature scaling. This dual-objective approach balances mimicking the teacher's nuanced knowledge with maintaining fidelity to true labels.

Two primary categories of knowledge transfer:

Response-Based Distillation

• Student mimics only the final output layer (logits) of the teacher
• Simpler to implement, computationally efficient
• Loses intermediate representational knowledge

Feature-Based Distillation

• Student learns to replicate the teacher's intermediate feature maps and attention patterns
• Uses hint layers and guided training to align internal representations
• Transfers richer structural knowledge about signal processing hierarchies
• Particularly effective for AMC tasks where intermediate RF feature extraction patterns are critical

Hybrid approaches combine both, using feature alignment in early layers and response matching in final layers.

Offline Distillation: The standard approach where a pre-trained teacher is frozen, and the student trains on the teacher's stored outputs. Requires one forward pass through the teacher per sample or pre-computed soft targets. Best when teacher training is complete and stable.

Online Distillation: Teacher and student are trained simultaneously, with the teacher's parameters updated alongside the student. The teacher can adapt to the student's learning progress. This includes co-distillation where multiple peer networks train together, each serving as both teacher and student.

Self-Distillation: A special case where the student and teacher share the same architecture, with deeper layers distilling knowledge into shallower ones of the same network during training.

Knowledge distillation is critical for deploying deep learning AMC models on software-defined radios and tactical edge devices. Key adaptations include:

• Teacher ensembles combining CNN, LSTM, and Transformer models trained on high-SNR synthetic data to capture diverse signal representations
• Student models distilled to run on FPGA fabric with INT8 quantization, achieving < 1ms inference latency
• Channel-robust distillation where the teacher is exposed to clean signals while the student trains on degraded, faded, and noisy versions
• Cross-receiver distillation transferring knowledge from a high-fidelity lab-grade receiver model to a student deployed on low-cost SDR hardware with different front-end characteristics

This enables state-of-the-art classification accuracy on devices with milliwatt power budgets.

The foundational two-model training paradigm where a large, pre-trained teacher network generates soft labels—probability distributions over classes rather than hard one-hot labels—that encode inter-class similarities. The compact student network is trained to match these soft targets, capturing the teacher's generalization capacity rather than merely memorizing training data.

• Soft targets contain dark knowledge about class relationships
• Student typically has 5-20x fewer parameters than teacher
• Training uses a weighted combination of soft and hard label losses
• Temperature parameter T in the final softmax controls softness of probability distribution

A complementary compression technique that reduces the numerical precision of model weights and activations from 32-bit floating point (FP32) to lower bit-width representations like 8-bit integers (INT8). Unlike knowledge distillation, quantization does not change the model architecture but reduces memory footprint and enables faster integer arithmetic on edge accelerators.

• INT8 quantization can reduce model size by 4x with minimal accuracy loss
• Quantization-aware training (QAT) simulates quantization during training for better accuracy
• Often applied after distillation for compound compression
• Critical for deployment on NPUs and FPGAs with integer-optimized pipelines

A sparsity-inducing technique that removes individual weights or entire neurons with magnitudes near zero, creating a sparse network with reduced computational requirements. Structured pruning removes entire channels or filters, enabling direct speedup on standard hardware, while unstructured pruning achieves higher compression but requires specialized sparse computation libraries.

• Magnitude-based pruning removes weights below a threshold
• Iterative pruning with fine-tuning recovers accuracy after removal
• Can achieve 50-90% sparsity with minimal performance degradation
• Often combined with distillation to transfer knowledge before pruning

An automated methodology for discovering optimal compact network architectures specifically designed for target hardware constraints. Rather than manually designing a student network, NAS algorithms explore a search space of possible architectures using reinforcement learning, evolutionary methods, or gradient-based optimization to find models that maximize the accuracy-efficiency tradeoff.

• Hardware-aware NAS incorporates latency and memory constraints directly into the search objective
• Can discover architectures that outperform hand-designed students
• Often used to generate the student architecture before distillation training
• Search spaces include kernel sizes, layer counts, and skip connection patterns

A matrix decomposition technique that approximates large weight matrices as products of smaller matrices, reducing both parameter count and computation. For a weight matrix W ∈ R^{m×n}, low-rank factorization represents it as W ≈ AB where A ∈ R^{m×r} and B ∈ R^{r×n} with rank r ≪ min(m,n).

• Singular Value Decomposition (SVD) identifies the optimal low-rank approximation
• Particularly effective for fully connected layers in modulation classifiers
• Can be applied post-training without retraining the full model
• Reduces both FLOPs and memory bandwidth requirements

The final deployment target for compressed modulation recognition models, where Field-Programmable Gate Arrays (FPGAs) provide reconfigurable hardware acceleration with deterministic low-latency inference. Knowledge distillation enables complex teacher models to be compressed into student architectures that fit within the limited DSP slices and block RAM of edge FPGAs.

• HLS (High-Level Synthesis) converts compressed models to hardware description languages
• Typical edge FPGAs have < 10W power envelopes for sustained field operation
• Streaming architectures process IQ samples with sub-millisecond latency
• Combined compression pipeline: distillation → quantization → FPGA synthesis