Knowledge Distillation

The foundational two-model paradigm where a pre-trained, high-capacity teacher network generates soft labels for a compact student network. The student learns not just the correct class, but the full probability distribution over all classes.

• Teacher: Large ResNet or Transformer with high accuracy
• Student: MobileNet or depthwise separable CNN for FPGA deployment
• Soft targets encode inter-class similarities that hard labels discard
• The student typically has 10-50x fewer parameters than the teacher

The teacher's output logits are divided by a temperature parameter (T) before the softmax function to soften the probability distribution. Higher temperatures reveal the dark knowledge of non-target class relationships.

• T=1: Standard softmax probabilities
• T>1: Softer distribution, exposes class similarities
• T→∞: Uniform distribution over all classes
• The student is trained with the same high temperature, but evaluated at T=1
• Typical temperature values range from 2 to 20 depending on the task

The student network is optimized using a weighted combination of two loss terms: the standard cross-entropy with ground-truth hard labels and the Kullback-Leibler divergence with the teacher's soft targets.

• L_total = α * L_soft + (1-α) * L_hard
• L_soft: KL divergence between student and teacher softened outputs
• L_hard: Cross-entropy with true labels (optional but stabilizes training)
• α typically set between 0.5 and 0.9, emphasizing soft targets
• Soft loss gradient scales by 1/T², requiring loss weighting by T²

The most common distillation paradigm where the student mimics the final output layer of the teacher. For modulation classification, this means matching the predicted probability distribution over modulation schemes like QPSK, 16-QAM, and 64-QAM.

• Directly transfers the teacher's decision boundary knowledge
• Computationally lightweight compared to feature-based methods
• Effective when teacher and student share similar architectural inductive biases
• Particularly useful for IQ sample classifiers with limited training data

Extends response-based methods by forcing the student to match the teacher's intermediate feature representations at specific layers. This transfers hierarchical knowledge about signal structure learned in early convolutional layers.

• Uses L2 or cosine similarity loss between teacher and student feature maps
• Requires careful selection of hint layers and guided layers
• May need linear projection layers to match differing channel dimensions
• Improves student generalization for complex modulation families like APSK

Transfers the structural relationships between data samples rather than individual outputs. The student learns to preserve the relative distances and angular similarities that the teacher observes across a batch of IQ samples.

• Distance-wise: Preserves Euclidean distances between sample embeddings
• Angle-wise: Preserves cosine similarities in the teacher's penultimate layer
• Instance relationship graphs capture higher-order sample interactions
• Robust to domain shift between training and deployment RF environments

Deploying a distilled student model on an FPGA-based spectrum analyzer enables continuous, low-latency classification of signals across wide bandwidths. The compact model fits entirely in on-chip Block RAM (BRAM) and utilizes DSP48 slices for parallel inference.

• Latency: Sub-millisecond classification per IQ frame
• Throughput: Processes 100+ MHz of instantaneous bandwidth
• Power: Operates within a <5W thermal envelope on mid-range FPGAs

This allows for persistent monitoring in remote or tactical environments where cloud connectivity is unavailable.

For signals intelligence (SIGINT) applications on small form-factor drones or man-portable units, a knowledge-distilled classifier running on a Xilinx Zynq UltraScale+ MPSoC provides on-device modulation recognition without RF emissions from data backhaul.

• Integrates with Vitis AI for optimized DPU acceleration
• Uses integer-only inference to eliminate floating-point hardware requirements
• Achieves >95% classification accuracy on 11 common modulation schemes at 10 dB SNR

The student model's reduced memory footprint allows co-residency with other mission-critical firmware on shared programmable logic.

A distilled neural network integrated directly into the digital front-end of a software-defined radio (SDR) enables autonomous, per-packet modulation recognition. The student model processes raw IQ samples as a streaming pipeline, classifying the modulation type before the demodulation stage.

• Implements a streaming architecture with ping-pong buffers for continuous operation
• Utilizes operator fusion to merge convolution, batch normalization, and activation layers into single kernels
• Enables dynamic switching of demodulation schemes without control-channel overhead

This tight integration reduces the radio's reaction time to changing signal environments.

Deploying identical distilled student models across a mesh of low-cost software-defined radio (SDR) nodes creates a distributed spectrum sensing fabric. Each node performs local modulation classification and shares only metadata, not raw IQ data, back to a central fusion center.

• Student models are compressed via post-training quantization (PTQ) to INT8 precision
• Cross-layer equalization minimizes accuracy loss from quantization
• Nodes operate on solar/battery power with <3W total system draw

The teacher-student paradigm ensures all edge nodes benefit from a centralized, high-capacity model trained on comprehensive datasets without requiring each node to store or run the large model.

In electronic warfare (EW) systems, a teacher model trained on an extensive library of known and simulated threat emitters distills its knowledge into a compact student for deployment on ruggedized VPX FPGA boards. The student performs real-time modulation identification to cue jamming or countermeasure responses.

• Supports open set recognition to flag unknown modulation types for operator review
• Uses hardware-aware NAS to discover architectures optimized for specific FPGA fabric
• Maintains classification robustness against adversarial perturbations via defensive distillation techniques

The distilled model's deterministic latency is critical for time-sensitive EW engagements where microsecond delays are unacceptable.

Within Open RAN architectures, a distilled modulation classifier deployed on the Near-Real-Time RAN Intelligent Controller (Near-RT RIC) identifies the modulation and coding scheme (MCS) of incoming uplink signals to inform dynamic resource allocation decisions.

• Student model runs as an xApp on the RIC platform
• Processes MIMO spatial streams with per-layer modulation identification
• Feeds classification results into the scheduler for optimized PRB allocation

Knowledge distillation allows the classifier to meet the strict <10ms control loop latency requirements of the Near-RT RIC while maintaining the accuracy of a much larger offline-trained model.

A compression technique that reduces the numerical precision of a pre-trained neural network's weights and activations without requiring retraining. PTQ converts 32-bit floating-point tensors to 8-bit integers, enabling faster inference on FPGA DSP slices. Unlike knowledge distillation, PTQ does not alter the model architecture—it only changes numerical representation. Common calibration methods include MinMax and KL divergence to minimize quantization error.

The process of removing redundant or low-magnitude connections from a neural network to reduce its memory footprint and computational complexity. Pruning can be unstructured (zeroing individual weights) or structured (removing entire channels/filters). When combined with knowledge distillation, a pruned student model can recover accuracy lost during sparsification by learning from the teacher's soft labels.

A method that simulates the effects of low-precision arithmetic during both forward and backward passes of training. Unlike PTQ, QAT allows the network to adapt its weights to compensate for quantization error. The Straight-Through Estimator (STE) approximates gradients through non-differentiable rounding operations. QAT is often applied to the student model after distillation to further compress it for FPGA deployment.

An automated methodology for discovering optimal neural network topologies that are inherently efficient for specific hardware targets. Hardware-aware NAS incorporates latency and energy feedback directly into the search objective. NAS can design compact student architectures from scratch, which are then trained via knowledge distillation to match the accuracy of a larger teacher model on modulation classification tasks.

A graph-level optimization that combines multiple consecutive neural network operations into a single computational kernel. Common fusions include Conv+BN+ReLU and MatMul+Bias+Activation. By eliminating intermediate memory writes, operator fusion reduces memory bandwidth bottlenecks. This technique is applied post-distillation to further accelerate the student model on FPGA streaming architectures.