Knowledge Distillation

The core framework involves a two-model setup. A high-capacity teacher model—often a large ensemble or deep neural network—is pre-trained on extensive RF data. A lightweight student model is then trained not on hard labels alone, but to replicate the teacher's output distribution. This transfers inductive biases and domain-invariant knowledge that the teacher has learned about hardware impairments, effectively compressing model size while preserving the ability to ignore channel artifacts.

Instead of training on one-hot encoded class labels, the student learns from soft targets—the teacher's probability distribution over all classes. A temperature parameter (T) is applied to the softmax function to soften this distribution, revealing the dark knowledge of inter-class similarities. For RF fingerprinting, this means the student learns not just that a signal belongs to Device A, but that Device A's signature is subtly similar to Device B's due to shared hardware components, creating a richer, more robust feature space.

The student model is optimized using a composite loss:

• Distillation Loss: Kullback-Leibler (KL) divergence between the softened student and teacher outputs, transferring dark knowledge.
• Student Loss: Standard cross-entropy between the student's hard predictions and ground-truth labels. A hyperparameter balances these two objectives. This dual-loss approach ensures the student learns both the teacher's nuanced decision boundaries and the absolute truth, preventing the propagation of teacher errors while maintaining channel-robust feature extraction.

Beyond output-level mimicry, this variant forces the student to match the teacher's intermediate feature representations. A regression loss minimizes the distance between the student's and teacher's activation maps at specific layers. In the context of channel-robust learning, this directly transfers the teacher's ability to disentangle device-specific features from channel-induced distortions at multiple levels of abstraction, ensuring the student's internal representations are equally invariant to multipath fading.

A technique where a model acts as its own teacher. A network is first trained conventionally. In a second phase, a copy of the same architecture is trained from scratch using the original model's soft predictions as targets. This process can be repeated iteratively. Self-distillation improves generalization and calibration without requiring a larger teacher, making it useful for refining channel-robust fingerprinting models when a pre-existing high-capacity teacher is unavailable.

In contrast to the traditional two-stage offline process, online distillation trains the teacher and student simultaneously in a single end-to-end process. Architectures like deep mutual learning allow a cohort of peer networks to teach each other. For channel-robust feature learning, this co-training dynamic can be coupled with domain adversarial objectives, allowing the peer networks to collaboratively discover and share domain-invariant features during training.

The foundational framework where a large, pre-trained teacher model generates soft labels—probability distributions over classes—that a smaller student model is trained to replicate. Unlike hard labels, soft labels capture the teacher's uncertainty and inter-class relationships, transferring dark knowledge that encodes channel-invariant feature representations. The student minimizes the Kullback-Leibler divergence between its softened output and the teacher's, often combined with the standard task loss.

A hyperparameter T applied to the softmax function that controls the softness of the teacher's output distribution:

• T=1: Standard softmax probabilities
• T>1: Produces softer probabilities, revealing more inter-class similarity
• Higher temperatures expose the dark knowledge embedded in the teacher's near-zero predictions
• The student uses the same temperature during training but reverts to T=1 at inference

An extension beyond output-level mimicry where the student learns to replicate the teacher's intermediate representations. The student is trained to minimize the distance between its hidden layer activations and the teacher's, often using:

• L2 loss between feature maps
• Maximum Mean Discrepancy (MMD) for distribution alignment
• Attention transfer to match spatial attention maps This is particularly effective for transferring channel-robust features learned in the teacher's deep layers.

A variant where the teacher and student share the same architecture, with the student being trained on the teacher's predictions. This can be done iteratively—each generation serves as the teacher for the next—or within a single training run using exponential moving average weight snapshots. Self-distillation has been shown to improve generalization and calibration without requiring a larger model, making it useful for refining channel-robust representations in resource-constrained deployments.

A dynamic paradigm where the teacher and student are trained simultaneously rather than sequentially. In architectures like deep mutual learning, multiple peer networks teach each other, or a single large model distills knowledge to a smaller one in parallel. This co-training approach allows the student to benefit from the teacher's evolving understanding of channel variations during training, rather than from a frozen snapshot.

Three distinct model compression strategies often used in combination:

• Knowledge Distillation: Trains a compact model to mimic a larger one's behavior
• Weight Pruning: Removes redundant connections from an existing network
• Post-Training Quantization: Reduces numerical precision (e.g., FP32 to INT8) Distillation is unique in that it can transfer learned invariances—such as channel robustness—that structural compression methods may not preserve.