Kullback-Leibler Divergence

KL divergence is always greater than or equal to zero. The minimum value of 0 is achieved if and only if the two distributions P and Q are identical almost everywhere. This property, proven via Gibbs' inequality, makes it a valid measure of divergence. In distillation, a KL loss approaching zero signals that the student's softmax output is perfectly matching the teacher's softened probability distribution.

KL divergence is not symmetric: D_KL(P ∥ Q) ≠ D_KL(Q ∥ P). This is its most critical departure from a true distance metric. The direction matters:

• Forward KL: D_KL(P ∥ Q) penalizes Q for ignoring regions where P has mass, leading to a 'mean-seeking' behavior.
• Reverse KL: D_KL(Q ∥ P) penalizes Q for placing mass where P has none, leading to a 'mode-seeking' behavior. In distillation, we typically compute D_KL(teacher ∥ student), using the teacher's distribution as the reference P.

KL divergence does not satisfy the triangle inequality. For three distributions P, Q, and R, it is not guaranteed that D_KL(P ∥ R) ≤ D_KL(P ∥ Q) + D_KL(Q ∥ R). This violation further distinguishes it from a proper metric like Euclidean distance. Consequently, KL divergence cannot be used to form a metric space, and geometric intuitions about distances in probability space can be misleading.

D_KL(P ∥ Q) measures the expected excess surprise when using Q to encode samples actually drawn from P. If an optimal code is designed based on distribution Q, but the true data comes from P, the KL divergence quantifies the average number of extra nats (or bits, if using log base 2) required per sample. This connects directly to cross-entropy: H(P, Q) = H(P) + D_KL(P ∥ Q). Minimizing cross-entropy is equivalent to minimizing KL divergence since H(P) is constant.

For D_KL(P ∥ Q) to be finite, P must be absolutely continuous with respect to Q. This means that for any event x, if Q(x) = 0, then P(x) must also equal 0. If P places probability mass on an event that Q declares impossible, the divergence becomes infinite. In neural network distillation with softmax outputs, this is practically avoided because temperature scaling ensures all class probabilities are strictly positive, preventing zero-probability mismatches.

KL divergence is jointly convex in the pair (P, Q). For any two pairs of distributions (P₁, Q₁) and (P₂, Q₂) and a mixing weight λ ∈ [0,1]:

• D_KL(λP₁ + (1-λ)P₂ ∥ λQ₁ + (1-λ)Q₂) ≤ λ D_KL(P₁ ∥ Q₁) + (1-λ) D_KL(P₂ ∥ Q₂) This convexity property is essential for optimization. When used as a distillation loss, it guarantees that gradient-based optimization converges to a global minimum when the student's parameterization forms a convex set with respect to the output probabilities.

A model compression technique where a smaller student model is trained to replicate the behavior and predictive performance of a larger, more complex teacher model. The student learns from the teacher's softened probability outputs, capturing dark knowledge about inter-class relationships that hard labels cannot provide. This process transfers generalization capabilities while dramatically reducing inference latency and memory footprint.

The probability distributions over classes produced by a teacher model, typically smoothed by a high temperature parameter. Unlike hard labels that assign 100% probability to the correct class, soft targets reveal the teacher's learned similarity structure:

• A cat image might have 0.7 probability for 'cat', 0.2 for 'dog', 0.1 for 'fox'
• This distribution encodes that cats are more similar to dogs than foxes
• The student learns these relative relationships, improving generalization

A hyperparameter T applied to the softmax function during distillation that controls the softness of the output probability distribution. Higher temperatures produce softer distributions:

• T=1: Standard softmax, sharp distribution
• T>1: Softens probabilities, revealing dark knowledge about class similarities
• T→∞: Approaches uniform distribution
• T→0: Approaches hard argmax The temperature is typically raised during teacher inference and student training, then set back to 1 for deployment.

A composite objective function combining two components:

1. Soft Target Loss: KL divergence between teacher and student soft targets, weighted by T² 2. Hard Target Loss: Standard cross-entropy against ground-truth labels

The weighted sum balances mimicking the teacher's internal representations against fitting the actual data. The α hyperparameter controls this tradeoff, typically set between 0.1 and 0.9, with higher values favoring the teacher's guidance.

The implicit information about the generalization and similarity structure of the data encoded in the relative probabilities of incorrect classes within a teacher model's softmax output. This knowledge is invisible in hard labels but critical for student performance:

• A 3 may be assigned 0.6 probability, but 8 gets 0.3 and 5 gets 0.05
• This reveals that 3 is visually more similar to 8 than to 5
• The student internalizes this geometry, learning smoother decision boundaries
• Dark knowledge is what makes distillation outperform training on original data alone

A training framework consisting of a high-capacity teacher model that generates supervisory signals for a compact student model to mimic. Key architectural considerations:

• Teacher is typically frozen during student training
• Student may share no architectural similarity with teacher
• The only requirement is that both produce comparable probability distributions
• Common pairings: BERT→BiLSTM, ResNet-50→MobileNet, ensemble→single model
• The student learns both the teacher's outputs and its internal feature representations