Knowledge Distillation

The foundational two-model setup where a large, pre-trained teacher network generates soft probability distributions over modulation classes, and a compact student network is trained to mimic these outputs rather than hard ground-truth labels. The teacher's logits are softened using a temperature parameter (T) in the softmax function, exposing the relative similarities between incorrect classes that the teacher has learned. This dark knowledge provides richer supervision than one-hot labels alone.

A mechanism that controls the softness of the teacher's output distribution. A temperature T > 1 flattens the probability distribution, revealing inter-class similarities that the teacher has internalized. For example, a teacher classifying a 16-QAM signal might assign significant probability mass to 64-QAM, teaching the student that these modulations share structural similarities. The student is trained with a composite loss function combining the distillation loss (matching soft targets) and the standard cross-entropy loss against true labels.

The most common distillation paradigm where only the final logit layer outputs are transferred from teacher to student. The student minimizes the Kullback-Leibler divergence between its softened output distribution and the teacher's. In AMC applications, this allows a student with 90% fewer parameters to retain the teacher's ability to distinguish between spectrally similar modulations like 64-QAM and 256-QAM at low SNR, where hard labels would obscure the decision boundary.

An advanced technique where the student learns to match the teacher's intermediate feature representations, not just final outputs. A hint layer in the teacher guides a guided layer in the student via an L2 regression loss. For I/Q signal processing, this forces the student to learn the same hierarchical feature extractors—from low-level spectral patterns to high-level modulation signatures—that the teacher discovered, leading to better generalization on unseen channel conditions.

• Offline Distillation: The standard approach where a pre-trained, frozen teacher supervises student training. Requires one teacher training run.
• Online Distillation: Teacher and student are trained simultaneously, with the teacher's parameters updated alongside the student's. This is useful when no pre-trained AMC model exists for a new frequency band.
• Self-Distillation: A single network acts as its own teacher, where deeper layers distill knowledge to shallower layers of the same architecture during training.

Distilled student models enable real-time modulation recognition on resource-constrained edge hardware such as software-defined radios and FPGA-based SIGINT platforms. A distilled AMC model can achieve inference latencies under 1 millisecond on embedded devices while maintaining classification accuracy within 1-2% of a GPU-scale teacher. This is critical for tactical electronic warfare systems where size, weight, and power (SWaP) constraints prohibit large model deployment.

A compression technique that reduces the numerical precision of a neural network's weights and activations—typically from 32-bit floating point to 8-bit integer—to decrease inference latency and memory footprint. Quantization is often applied to a distilled student model as a secondary compression step.

• Post-training quantization (PTQ) requires no retraining
• Quantization-aware training (QAT) simulates low-precision during distillation
• INT8 inference can yield 4x memory reduction with minimal accuracy loss

The foundational training paradigm where a compact student network learns to replicate the behavior of a larger, high-capacity teacher network. The student is trained on the teacher's softened output probabilities—not just hard labels—capturing inter-class relationships that raw labels miss.

• Teacher provides soft targets via a temperature-scaled softmax
• Student can have a completely different architecture from the teacher
• Effective even when the student has 10x fewer parameters

A compression method that removes redundant or low-magnitude connections from a neural network. When combined with knowledge distillation, pruning can be applied to the student model post-training to achieve further compression.

• Unstructured pruning zeros out individual weights, creating sparse matrices
• Structured pruning removes entire channels or filters for hardware-friendly speedups
• Iterative prune-and-fine-tune cycles preserve distilled knowledge

A self-supervised training method that learns robust signal representations by pulling augmented views of the same I/Q sample together in the embedding space while pushing apart views from different samples. Contrastive pre-training produces rich feature extractors that serve as excellent teacher models for subsequent distillation.

• No labeled data required for pre-training
• Representations capture signal structure not just class boundaries
• Combined with distillation for state-of-the-art few-shot AMC

Neural network architectures that natively process complex-valued I/Q data using complex weights and activation functions, preserving phase information often lost in real-valued decomposition. Distilling a complex-valued teacher into a real-valued student is an active research area for edge deployment.

• Preserves phase relationships critical for modulation recognition
• Complex batch normalization and complex ReLU activations
• Student may use real-valued approximations for hardware compatibility