Knowledge Distillation

Instead of training on hard one-hot labels, the student model learns from the teacher's softmax output probabilities (soft labels). These probability vectors encode rich inter-class relationships—for example, that a '3' is more similar to an '8' than to a 'cat'. This dark knowledge transfers generalization without exposing raw training data. The temperature parameter T in the softmax function controls the smoothness of the output distribution:

• Higher T produces softer probabilities, revealing more subtle class similarities
• Lower T approaches hard labels, reducing information transfer
• Typical values range from T=2 to T=20 during distillation

Defensive distillation specifically hardens models against inversion and adversarial attacks by training a second model on the smoothed predictions of the first. The key mechanism: the distilled model's decision surface is flattened around training points, reducing the gradient information exploitable by attackers.

• The student model's loss surface exhibits smaller curvature near training samples
• Gradient-based inversion attacks produce blurrier, less recognizable reconstructions
• Membership inference attack AUC drops significantly on distilled models
• Originally proposed by Papernot et al. (2016) for adversarial robustness, later extended to privacy

Private Aggregation of Teacher Ensembles (PATE) combines knowledge distillation with differential privacy. An ensemble of teacher models is trained on disjoint partitions of sensitive data, and a student model learns from their noisy aggregated votes.

• Each teacher sees only a subset of data, limiting per-model exposure
• The aggregation step adds calibrated Laplacian or Gaussian noise to vote counts
• The student never accesses raw data, only the privacy-guaranteed labels
• Provides provable (ε, δ)-differential privacy guarantees
• Effective for tasks where unlabeled public data is available for student training

Knowledge distillation naturally implements an information bottleneck—the student model can only learn what the teacher's output probabilities convey. Fine-grained details about individual training samples are compressed away, leaving only class-level discriminative features.

• Mutual information between student representations and individual inputs is reduced
• The student learns invariant representations focused on task-relevant patterns
• This compression is measured by the decrease in mutual information I(X; Z) between inputs X and latent representations Z
• The bottleneck width is controlled by the student's architecture capacity and the distillation temperature

The distillation temperature T directly controls the privacy-utility trade-off. Higher temperatures produce softer labels that mask individual sample contributions but may lose task-critical information.

• Low T (1-2): Near-hard labels, minimal privacy benefit, high utility preservation
• Medium T (5-10): Balanced trade-off, class relationships preserved, individual traces obscured
• High T (20+): Strong privacy through maximum smoothing, risk of utility degradation
• Optimal T depends on dataset size, number of classes, and sensitivity requirements
• Cross-validation on a held-out public dataset helps calibrate T

The student model's reduced capacity acts as a structural defense. A smaller network simply lacks the parameters to memorize individual training examples, forcing it to learn only compressed, generalized patterns.

• Parameter count reduction: Typical student models use 10-50x fewer parameters than teachers
• Width and depth constraints: Narrower layers and fewer layers limit representational capacity
• Bottleneck layers: Intentional low-dimensional latent spaces force aggressive compression
• Quantization: Reduced-precision weights further limit information storage
• The capacity gap between teacher and student bounds the maximum information leakage

A defensive technique where a second student model is trained on the softmax output vectors of a first teacher model, using a higher temperature parameter. This smooths the decision surface, masking the gradient information typically exploited by adversarial example and model inversion attacks. The student learns class probabilities rather than hard labels, reducing the signal available for reconstructing training data.

Private Aggregation of Teacher Ensembles trains a student model on the noisy, aggregated votes of an ensemble of teacher models, each trained on disjoint partitions of sensitive data. Key properties:

• Adds Gaussian noise to vote tallies to enforce differential privacy
• Only the student model is publicly released, never the teachers
• Provides provable privacy budget (epsilon) guarantees
• Enables knowledge transfer without exposing individual training records

An attack that reconstructs representative samples of a target class by exploiting access to a trained model's confidence scores and parameters. The attacker iteratively optimizes a random input to maximize the model's prediction confidence for a specific class, effectively reversing the learned mapping. Knowledge distillation can mitigate this by training students on smoothed soft labels rather than high-confidence predictions that leak fine-grained input details.

A theoretical framework that formalizes the trade-off between compression and prediction. The objective minimizes mutual information between the input and a latent representation while maximizing mutual information with the target task. This naturally limits inversion risk by discarding input details irrelevant to the task. Distillation implements a practical form of this bottleneck by forcing the student to learn only what the teacher deems task-relevant.

A lightweight defense that truncates a model's prediction vector to return only the top-k class labels instead of full probability distributions. By hiding the confidence scores for non-predicted classes, the information available for membership inference and model inversion attacks is severely limited. This technique is often combined with distillation, where the student is trained to output only essential classification information.

A distributed training paradigm where a model is partitioned between a client and server. The client processes raw data through initial layers and shares only intermediate activations (smashed data) rather than raw inputs or full gradients. The server completes the forward pass and backpropagates gradients only to the cut layer. This reduces the attack surface for gradient inversion and complements distillation by keeping sensitive data local.