Knowledge Distillation

The teacher model is a large, high-capacity network (often an ensemble) trained to convergence. The student model is a compact architecture with fewer parameters. The student learns to mimic the teacher's output distribution rather than training directly on hard labels.

• Teacher provides rich dark knowledge about inter-class similarities
• Student architecture can be heterogeneous (different from teacher)
• Common pairings: BERT → TinyBERT, ResNet-152 → ResNet-18

A temperature parameter (T) softens the teacher's output logits before the softmax function. Higher T produces a softer probability distribution that reveals the teacher's learned similarities between classes.

• T=1: Standard softmax probabilities
• T>1: Flattens distribution, exposing class relationships
• Student trains on these soft targets rather than hard one-hot labels
• At inference, student uses T=1 for crisp predictions

The student optimizes a weighted combination of two loss terms. The distillation loss matches the student's softened outputs to the teacher's softened outputs using KL divergence. The student loss uses standard cross-entropy with ground truth labels.

• L_total = α * L_soft + (1-α) * L_hard
• α balances imitation vs. ground truth adherence
• KL divergence: L_soft = T² * KL(teacher_probs || student_probs)
• T² scaling compensates for gradient magnitude reduction at high temperatures

Dark knowledge refers to the information encoded in the relative probabilities of incorrect classes. A teacher classifying a car image might assign 0.001 to 'truck' but 0.000001 to 'apple', revealing that trucks are more similar to cars than apples are.

• Captures generalization patterns not present in hard labels
• Encodes the teacher's learned feature manifold
• Provides richer supervision signal per training example
• Enables student to generalize better than training on labels alone

Beyond output probabilities, the student can learn to match the teacher's intermediate representations. The student's hidden layer activations are trained to align with the teacher's feature maps through regression or contrastive losses.

• Hint layers: Teacher layers selected as supervision targets
• Guided layers: Student layers trained to mimic hints
• Uses L2 loss or attention transfer between feature maps
• Effective when teacher and student share similar architectural patterns

Instead of matching individual examples, the student learns to preserve the relationships between examples as encoded by the teacher. This transfers structural knowledge about the data manifold.

• Instance relationship graphs: Distances between sample embeddings
• Similarity-preserving loss: Maintains pairwise similarity matrices
• Contrastive representation distillation: Maximizes mutual information
• Robust to architectural differences between teacher and student

An attack where an adversary systematically queries a target model's prediction API to extract a functionally equivalent copy. Unlike distillation, which is a legitimate compression technique, model stealing is adversarial. The attacker uses the teacher's soft labels or hard predictions to train a clone model, effectively bypassing the need for the original proprietary dataset. This is a direct threat to Model-as-a-Service (MaaS) business models.

The foundational framework for knowledge distillation. A large, high-capacity teacher model generates soft targets—probability distributions over classes that capture inter-class similarities. A compact student model is trained to mimic these soft targets, often using a temperature parameter (T) in the softmax function to soften the probability distribution. This transfers the teacher's generalization ability and dark knowledge to the student.

A mathematical framework providing provable privacy guarantees against extraction attacks, including those leveraging distillation. By adding calibrated noise to the teacher's training process or output, DP ensures that the student model cannot memorize or reveal individual training records. PATE (Private Aggregation of Teacher Ensembles) is a specific architecture that combines knowledge distillation with differential privacy to train models with strong privacy bounds.

The broader machine learning paradigm where knowledge gained from solving one problem is applied to a different but related problem. Knowledge distillation is a specific form of transfer learning where the knowledge source is a pre-trained teacher model rather than a pre-existing dataset. The student learns the teacher's inductive bias and feature representations, often achieving higher accuracy than training directly on the original hard labels alone.

A security technique originally proposed to defend neural networks against adversarial examples. A model is trained at a high temperature to produce soft labels, and a second model is distilled from it. The resulting model's smoother decision surface reduces the effectiveness of small input perturbations designed to cause misclassification. While later shown to be circumventable by stronger attacks, it remains a foundational concept in adversarial robustness research.