Knowledge Distillation

The core mechanism involves a teacher ensemble (large, high-capacity models) generating soft labels—probability distributions over classes smoothed by a high temperature parameter (T). A compact student model is trained to mimic these soft targets rather than hard ground-truth labels. The softened distribution reveals the teacher's learned similarity structure between classes (e.g., 'a 3 looks more like an 8 than a 1'), transferring dark knowledge that encodes generalization patterns while discarding instance-specific overfitting artifacts that enable membership inference.

The student optimizes a composite objective combining two terms:

• Soft target loss: Kullback-Leibler divergence between the student's softened logits and the teacher's softened outputs, weighted by T²
• Hard target loss: Standard cross-entropy with ground-truth labels (optional, smaller weight) The temperature scaling during training compresses the dynamic range of probabilities, forcing the student to learn the teacher's uncertainty calibration rather than memorizing high-confidence predictions on individual training samples. This calibration transfer is what incidentally reduces prediction entropy gaps between members and non-members.

Knowledge distillation reduces membership inference vulnerability through three interconnected effects:

• Ensemble smoothing: Teacher averaging cancels out instance-specific noise and memorization spikes that individual models encode
• Soft target regularization: Training on continuous probability vectors rather than discrete labels prevents the student from learning brittle, high-confidence decision boundaries around individual training points
• Capacity bottleneck: The student's deliberately constrained architecture cannot encode exact replicas of training data, functioning as an information bottleneck that preserves class-level structure while discarding instance-level artifacts The result is a model that generalizes well but resists exposure metric extraction.

The temperature parameter (T) directly controls the privacy-utility tradeoff in distillation:

• T = 1: Standard softmax, minimal privacy benefit
• T = 2–5: Moderate smoothing, reduces confidence gap between training and non-training samples
• T = 10–20: Heavy smoothing, strong privacy amplification but risk of utility degradation Higher temperatures flatten the teacher's output distribution, making it harder for an attacker to detect the subtle prediction entropy differences that membership inference classifiers exploit. This connects distillation to differential privacy principles—temperature scaling functions analogously to calibrated noise injection, though without formal epsilon guarantees.

Knowledge distillation is the core mechanism within the Private Aggregation of Teacher Ensembles (PATE) framework, which achieves formal differential privacy guarantees:

• Multiple teachers are trained on disjoint data partitions
• Teachers vote on labels with Gaussian noise added to the vote counts
• A student is distilled from the noisy aggregated outputs
• Only the student is publicly released; teachers remain private This architecture demonstrates how distillation's inherent privacy properties can be amplified and formalized through privacy amplification by subsampling and explicit noise mechanisms, achieving provable membership inference resistance.

Despite its privacy benefits, distilled models remain vulnerable to certain attacks:

• Label-only attacks: Even without confidence scores, an attacker can exploit the student's distilled decision boundaries if they overfit to teacher quirks
• Teacher memorization transfer: If the teacher ensemble itself memorized training data, the student may inherit these artifacts through the soft labels
• Distillation temperature inversion: Attackers with knowledge of the distillation process can reverse the temperature scaling to recover sharper signals Mitigation requires combining distillation with adversarial regularization, gradient clipping, and formal privacy budget accounting for defense-in-depth.

The foundational two-model setup where a large, high-capacity teacher model generates soft label distributions used to train a compact student model. The student learns to mimic not just the correct class, but the full probability distribution over all classes, capturing the teacher's learned similarities between categories. This transfer of dark knowledge—the relative probabilities of incorrect classes—provides richer supervision than hard labels alone.

A temperature parameter (T) is applied to the teacher's final softmax layer to soften the output probability distribution. Higher temperatures flatten the distribution, revealing the dark knowledge embedded in the relative probabilities of non-target classes. The student is trained using a weighted combination of the standard hard-label cross-entropy loss and the KL divergence between its own softened outputs and the teacher's softened outputs, balancing task accuracy with knowledge transfer.

Knowledge distillation enables efficient machine unlearning by retraining only the compact student model rather than the full teacher ensemble. When a data deletion request arrives, the affected teacher model is retrained on the remaining partition, and the student is re-distilled from the updated ensemble. This sharded retraining approach dramatically reduces the computational cost of compliance with data erasure regulations like GDPR's right to be forgotten.

Distilled models exhibit reduced prediction entropy gaps between training and non-training samples—the primary signal exploited by membership inference attacks. Because the student learns from smoothed, interpolated soft targets rather than memorized hard labels, it generalizes more uniformly across seen and unseen data. Empirical studies show distilled models can reduce membership inference attack AUC by 5-15% compared to equivalently sized models trained directly on private data.