Knowledge Distillation Defense

The student model is trained exclusively on the soft labels (probability distributions) produced by the teacher on a clean, trusted dataset. Unlike hard labels, soft labels encode rich inter-class similarity knowledge. The defense hypothesis is that backdoor triggers corrupt the teacher's soft labels only when the trigger is present; on clean data, the teacher's benign knowledge dominates, allowing the student to learn correct decision boundaries without inheriting the backdoor mapping.

A hyperparameter T (temperature) is applied to the teacher's output logits before the softmax operation. Higher temperatures produce softer probability distributions that reveal more granular inter-class relationships. During distillation defense, elevated temperature smooths out potential adversarial artifacts in the teacher's output while preserving the structural knowledge needed for the student to generalize correctly on clean inputs.

The defense critically depends on access to a verified clean dataset that is representative of the original training distribution but guaranteed to contain no poisoned or triggered samples. This dataset serves as the transfer medium. If the clean set is too small or lacks sufficient class diversity, the student may fail to capture the full benign knowledge from the teacher, resulting in degraded standard accuracy.

The core mechanism exploits the fact that backdoor triggers function as shortcuts in the teacher's feature space. When the trigger is absent from all distillation inputs, the teacher never activates its backdoor pathway. The student therefore never observes the trigger-to-target-label mapping and cannot learn it. The backdoor is effectively pruned by omission during the knowledge transfer process.

The student model can use a different architecture than the teacher, often a smaller, more efficient network. This architectural mismatch can further disrupt backdoor transfer because the student's reduced capacity may lack the representational space to encode both the primary task and the hidden trigger association. Common choices include MobileNet or compact ResNet variants distilled from larger teacher ensembles.

The defense is not foolproof. If the backdoor trigger is feature-level rather than input-space (e.g., a specific texture or frequency pattern present in clean data), the teacher may produce compromised soft labels even on the clean set. Additionally, if the clean dataset is inadvertently contaminated or the teacher's benign accuracy is low, the student inherits degraded performance. Adaptive attacks that design triggers to survive distillation remain an active research area.

A backdoor detection and mitigation technique that reverse-engineers potential triggers by finding the minimal perturbation required to misclassify all samples to a target label. It operates by solving an optimization problem: for each class, it searches for the smallest universal patch that causes misclassification. Once triggers are identified, the model can be patched by removing neurons associated with the backdoor behavior. Neural Cleanse is often used as a precursor step before applying Knowledge Distillation Defense to verify that the teacher model contains a backdoor.

A defense that removes backdoors by pruning dormant neurons that are activated by the trigger but remain inactive on clean validation data. The process involves:

• Recording neuron activations on clean samples
• Identifying neurons that are consistently dormant
• Pruning those neurons from the network
• Fine-tuning the pruned model on a clean dataset to recover accuracy Fine-Pruning and Knowledge Distillation Defense are complementary: pruning surgically removes trigger pathways, while distillation transfers only benign knowledge to a fresh architecture.

A detection method that separates clean and poisoned training data by clustering the activations of the final hidden layer for each class. Poisoned samples with backdoor triggers produce distinct activation patterns that form a separate cluster from clean samples. Key steps:

• Extract feature representations from the penultimate layer
• Apply dimensionality reduction (PCA or t-SNE)
• Cluster activations within each class
• Flag outlier clusters as potentially poisoned This technique helps identify which training samples to exclude before using the clean subset for distillation.

A detection method that identifies poisoned training examples by analyzing the singular value decomposition (SVD) of feature representations. The technique exploits the observation that backdoor triggers create a statistically separable signal in the spectrum of the covariance matrix of learned representations. Poisoned samples project strongly onto the top singular vector of a class, revealing themselves as outliers. Removing these flagged samples creates a clean dataset suitable for training a student model via Knowledge Distillation Defense.

A class of Byzantine-resilient algorithms used in federated learning to combine model updates from multiple clients while mitigating the impact of malicious contributions. These algorithms—including Krum, Trimmed Mean, and Median—operate on the principle that benign updates cluster together while poisoned updates are statistical outliers. Robust Aggregation defends against poisoning at the gradient level, whereas Knowledge Distillation Defense operates at the model level, making them complementary layers in a defense-in-depth strategy for distributed learning systems.

A technique for constructing a provably robust classifier by adding random Gaussian noise to inputs and returning the most probable prediction under that noise distribution. The method provides a mathematical guarantee that the prediction remains constant within a certified L2-radius around any input. While Knowledge Distillation Defense empirically removes backdoors, Randomized Smoothing offers formal guarantees against trigger-based attacks. The trade-off is that smoothing introduces inference-time randomness and may reduce clean accuracy.