Knowledge Distillation

The core framework involves a two-model setup: a large, pre-trained teacher network with high capacity and a smaller, lightweight student network. The student is trained not only on the true hard labels but also on the soft labels (class probabilities) produced by the teacher. These soft labels contain rich information about inter-class similarities—for example, indicating that 16-QAM is more similar to 64-QAM than to BPSK—which provides a stronger supervisory signal than one-hot encoded ground truth alone.

The teacher's output logits are passed through a softened softmax function using a temperature parameter (T). A higher temperature produces a softer probability distribution over classes, revealing the dark knowledge the teacher has learned about class relationships.

• T=1: Standard softmax, output is sharp and similar to hard labels.
• T>1: Softens the distribution, exposing secondary class probabilities.
• The student is trained to match this softened distribution, often using Kullback-Leibler divergence as the loss function.

The student's total loss function is a weighted combination of two objectives:

• Distillation Loss: The KL divergence between the softened teacher outputs and the softened student outputs, scaled by T² to maintain gradient magnitudes.
• Student Loss: Standard cross-entropy between the student's hard predictions and the ground-truth labels.

A hyperparameter α balances these two terms: L_total = α * L_distillation + (1-α) * L_student. This dual-objective ensures the student learns both the correct answers and the nuanced reasoning of the teacher.

Beyond matching final output probabilities, distillation can occur at the level of intermediate representations. The student is trained to mimic the teacher's internal feature maps or attention patterns.

• Hint-based training: The student's middle layers are regressed to match the teacher's corresponding hidden representations.
• Attention transfer: The student learns to replicate the teacher's spatial or channel-wise attention maps.
• This is particularly effective for signal processing tasks where the teacher learns robust feature extractors for IQ sample embeddings.

In few-shot modulation recognition, knowledge distillation serves a dual purpose. A teacher trained on abundant base classes transfers its inductive biases and feature extraction capabilities to a student. The student can then leverage this inherited knowledge to rapidly adapt to novel modulation types with only a handful of examples.

• The teacher's soft labels effectively augment the scarce support set by providing richer per-sample information.
• This reduces overfitting and improves the N-way K-shot classification accuracy for rare or emerging signal types.

Distillation is not limited to a static two-phase process. Advanced variants include:

• Online distillation: Teacher and student are trained simultaneously, with the teacher updating from the student's feedback in a mutual learning loop.
• Self-distillation: A single network distills knowledge from its own deeper layers to its shallower layers, or from later training epochs to earlier snapshots.
• Co-distillation: Multiple peer networks train collaboratively, each serving as both teacher and student to the others, improving ensemble robustness without additional inference cost.

The foundational two-model framework where a large, high-capacity teacher network generates supervisory signals for a compact student network. In few-shot modulation recognition, the teacher is typically pre-trained on abundant synthetic or base-class data, capturing rich inductive biases about signal structure. The student learns not just from hard labels but from the teacher's softened output distribution, which encodes inter-class similarities—for example, revealing that 16-QAM is structurally closer to 64-QAM than to BPSK. This relational knowledge is critical when the student must generalize from only K examples per novel modulation class.

A mechanism where the teacher's pre-softmax logits are divided by a temperature parameter T > 1 before producing soft labels. Higher temperatures smooth the probability distribution, exposing dark knowledge—the relative probabilities the teacher assigns to incorrect classes. For modulation classification, a soft label might show that a QPSK signal has a 0.15 probability of being confused with OQPSK under low SNR, teaching the student about signal constellation ambiguities without requiring explicit augmentation. During student training, the loss combines the soft target KL-divergence with standard hard-label cross-entropy.

The composite objective that trains the student to mimic the teacher. The standard formulation combines:

• KL-divergence loss between student and teacher softened outputs, weighted by α
• Cross-entropy loss against ground-truth hard labels, weighted by (1-α)
• Optional feature-level distillation where the student's intermediate representations are regressed toward the teacher's hidden activations

For few-shot RF tasks, feature distillation is particularly valuable—it transfers the teacher's learned invariance to channel impairments like frequency offset and fading, which the student would otherwise struggle to learn from limited examples.

A variant where the teacher and student share the same architecture, with the student training on its own softened predictions from previous epochs or from deeper layers. In few-shot modulation learning, self-distillation acts as a powerful regularizer: the model's own aggregated predictions across training augmentations provide a consistency target that reduces overfitting to the small support set. This technique is especially effective when combined with transductive inference, where the model refines its predictions by considering the entire query set distribution during distillation.

A compression paradigm where the student is trained without access to the original training data, using only the teacher model. The teacher's learned distribution is sampled via generative replay—synthesizing realistic IQ samples or feature vectors that maximize the teacher's response. For defense and spectrum monitoring applications, this enables deploying compact modulation classifiers on edge hardware without transferring sensitive signal databases. The student inherits the teacher's decision boundaries for known modulations while remaining small enough for FPGA or microcontroller deployment.

Two operational modes for knowledge transfer:

• Offline distillation: The teacher is pre-trained and frozen; the student learns from its static outputs. Standard for few-shot modulation where the teacher is trained once on abundant base classes.
• Online distillation: Teacher and student train simultaneously, with bidirectional knowledge exchange. In mutual learning variants, a pool of students co-train, sharing soft targets. For dynamic spectrum environments, online distillation enables continuous adaptation—as new modulation types appear, the ensemble collectively refines its understanding without catastrophic forgetting of previously learned schemes.