Soft Targets

A soft target is a vector of probabilities produced by applying the softmax function to a teacher model's output logits. Each element represents the model's confidence that the input belongs to a specific class. For example, for an image of a cat, a model might output [cat: 0.7, dog: 0.25, lion: 0.05]. This distribution is inherently non-binary and non-sparse, containing more information than a one-hot label like [1, 0, 0].

The term 'dark knowledge' refers to the implicit, learned similarities between classes encoded within the soft targets. A model trained to distinguish 'cats' from 'dogs' learns that a 'cat' is more similar to a 'lion' than to a 'truck'. This inter-class similarity information is captured in the non-zero probabilities for incorrect classes. By learning to match these softened distributions, the student model internalizes the teacher's nuanced understanding of the feature space, often leading to better generalization than training on hard labels alone.

To control the 'softness' of the target distribution, a temperature parameter (T) is introduced into the softmax function: softmax(z_i / T).

• High Temperature (T > 1): Smoothens the distribution, making probabilities more uniform and amplifying dark knowledge. This is typically used during distillation.
• Low Temperature (T = 1): Recovers the standard softmax, producing a sharper distribution.
• Very Low Temperature (T → 0): Approximates a hard, one-hot label. Temperature scaling is essential for making the rich information in the logits accessible for the student model to learn from.

The core objective in logit-based knowledge distillation is to minimize the difference between the teacher's soft targets and the student's softened outputs. This is typically measured using Kullback-Leibler Divergence (KL Divergence), a statistical distance between two probability distributions. The total loss for the student is often a weighted sum: L_total = α * L_CE(hard_labels, student) + (1-α) * T^2 * L_KL(teacher_soft_targets, student_soft_outputs) Where L_CE is the standard cross-entropy loss with true labels, and α balances the two objectives.

Understanding the distinction is crucial for effective distillation.

Hard Targets (One-Hot Labels):

• Format: Sparse vector (e.g., [0, 1, 0, 0]).
• Information: Only specifies the correct class.
• Gradient Signal: Provides a single, strong signal for the correct class.
• Use Case: Standard supervised training.

Soft Targets (Teacher Distributions):

• Format: Dense probability vector (e.g., [0.05, 0.8, 0.1, 0.05]).
• Information: Specifies correct class and relative similarity to other classes.
• Gradient Signal: Provides many smaller, corrective signals across all classes.
• Use Case: Knowledge distillation, label smoothing.

Soft targets are a foundational concept with connections to several advanced methods:

• Label Smoothing: A regularization technique that manually creates soft targets by mixing the one-hot label with a uniform distribution. It can be seen as distilling knowledge from a simplistic 'teacher'.
• Data-Free Distillation: Techniques that generate synthetic data or leverage the teacher's own representations to produce soft targets when the original training data is unavailable.
• Online Distillation: A paradigm where the teacher model is not static but is updated concurrently with the student, meaning the soft targets are dynamically evolving during training.
• Self-Distillation: A model distills knowledge from its own deeper layers or earlier training checkpoints, using self-generated soft targets.

Knowledge distillation is a model compression technique where a smaller, more efficient student model is trained to mimic the behavior and outputs of a larger, more powerful teacher model. The process transfers the teacher's learned representations, including the rich dark knowledge in its soft targets, to the compact student.

• Primary Goal: Create a deployable model with minimal performance drop.
• Core Mechanism: Uses a distillation loss (e.g., KL Divergence) to align student outputs with teacher outputs.
• Applications: Enables deployment of large models (e.g., BERT, GPT) on mobile devices and edge hardware.

The teacher-student framework is the foundational architecture for knowledge distillation. It consists of a pre-trained, often cumbersome teacher model and a lightweight, trainable student model.

• Teacher Role: Provides supervisory signals beyond simple class labels, typically via its logits or soft targets.
• Student Role: Learns to approximate the teacher's function through a combined loss of distillation loss and standard cross-entropy loss.
• Key Insight: The student learns the teacher's generalization behavior, often achieving better performance than if trained on hard labels alone.

Temperature scaling is a technique used to soften the teacher model's output probability distribution, making the soft targets richer and easier for the student to learn from. A temperature parameter (T) is applied to the softmax function.

• Mechanism: softmax(logits / T). Higher T values produce a softer, more uniform probability distribution.
• Purpose: Amplifies the dark knowledge—the relative differences between non-target classes that are suppressed in a standard (T=1) softmax.
• Training: The same T is used when computing the distillation loss for the student. At inference, T is set back to 1.

Dark knowledge refers to the implicit, inter-class relationships and similarities captured within a trained neural network's softened output probabilities. This information, not present in one-hot labels, is the key value transferred via soft targets.

• Source: A model's confidence that a 'cat' image also somewhat resembles a 'lynx' or 'tiger'.
• Transfer: By learning to match these softened probabilities, the student model internalizes a more nuanced understanding of the data manifold.
• Analogy: It is the 'how' and 'why' behind a prediction, not just the final 'what' of a hard label.

Logits distillation is a direct form of knowledge distillation where the student model is trained to match the raw, pre-softmax output logits of the teacher model, rather than its post-softmax probabilities.

• Advantage: Avoids the potential information loss from the softmax saturation. Gradients can flow more directly.
• Loss Function: Typically uses Mean Squared Error (MSE) between teacher and student logits.
• Relation to Soft Targets: Logits can be seen as the unnormalized precursors to soft targets. Applying temperature scaling to logits before a softmax generates the softened distributions used in standard distillation.

Kullback-Leibler divergence is a statistical measure of how one probability distribution diverges from a second, reference distribution. It is the most common distillation loss function for matching soft targets.

• Role in Distillation: Minimizes the KL Divergence between the student's softened output distribution and the teacher's softened output distribution.
• Formula: KL(P_teacher || P_student) = Σ P_teacher * log(P_teacher / P_student). It is asymmetric.
• Interpretation: Measures the extra information (in nats or bits) required to encode samples from the teacher's distribution using the student's distribution. A lower KL divergence indicates a better match.