Soft Targets

Unlike hard targets (one-hot encoded vectors), soft targets are continuous probability distributions over all possible classes. Each class is assigned a probability score, typically generated by applying a softmax function to the teacher model's logits. For example, an image of a 'cat' might yield probabilities like: cat: 0.85, dog: 0.12, fox: 0.03. This format is the fundamental data structure for transferring dark knowledge.

The richness of a soft target is controlled by a temperature parameter (T) applied within the softmax function. A higher temperature (e.g., T > 1) produces a 'softer,' more uniform distribution, amplifying the small probabilities of non-target classes. This makes the inter-class relationships more pronounced and easier for the student to learn. A lower temperature (T = 1) yields a distribution closer to the original model confidence. The optimal T is a hyperparameter tuned for the specific distillation task.

The primary value of soft targets lies in the dark knowledge they encode. This refers to the implicit information about the similarity and relationships between different classes, as learned by the teacher model. For instance, a teacher trained on ImageNet knows that a 'Persian cat' is more similar to a 'Maine Coon' than to a 'school bus'. This relational information, embedded in the probability scores of non-target classes, provides a superior learning signal compared to a simple correct/incorrect label, leading to better student generalization.

In the standard knowledge distillation loss function, soft targets serve as the primary supervisory signal for the student. The total loss is often a weighted sum of two components:

• Distillation Loss: Measures the discrepancy (e.g., using Kullback-Leibler Divergence) between the student's softened outputs and the teacher's soft targets.
• Hard Label Loss: The standard cross-entropy loss with the ground-truth labels. This combined objective allows the student to learn both the fine-grained class similarities from the teacher and the definitive task boundaries from the original data.

It is crucial to distinguish soft targets from logits. Logits are the raw, unnormalized scores output by the final linear layer of a neural network. Soft targets are the normalized probability distribution derived from these logits via the (temperature-scaled) softmax. While logits distillation is a valid technique, soft targets are generally more stable for training as they are bounded between 0 and 1. The choice between distilling logits or probabilities depends on the specific distillation framework and loss function used.

While foundational to classification tasks, the concept of soft targets extends to other domains:

• Sequence Generation: In sequence-level distillation, the teacher's distribution over entire output sequences (e.g., in machine translation) acts as a soft target.
• Object Detection: Teacher model's softened class probabilities and softened bounding box regressions can be distilled.
• Reinforcement Learning: The value function or policy distribution of a teacher agent can serve as a soft target for a student. This demonstrates the versatility of probabilistic supervision for compressing models across diverse AI tasks.

Knowledge Distillation is the overarching model compression technique where a smaller student model is trained to mimic the predictive behavior of a larger, more complex teacher model. The core mechanism involves using the teacher's output—often as Soft Targets—as a richer training signal than standard one-hot labels. This process transfers not just the correct class prediction but also the teacher's learned understanding of inter-class relationships, enabling the student to achieve higher accuracy than if trained on hard labels alone.

Dark Knowledge is the implicit, relational information encoded within a trained model's softened output distribution. It represents the model's learned understanding of similarities between different classes—for example, that a picture of a 'cat' is more similar to a 'lynx' than to an 'airplane'. This information, which is absent in one-hot labels, is the valuable signal captured in Soft Targets and transferred during knowledge distillation. The term highlights that this nuanced understanding is a hidden byproduct of training that can be explicitly leveraged to teach a more efficient model.

Temperature Scaling is a critical technique used to generate effective Soft Targets. It involves dividing the logits (pre-softmax outputs) of the teacher model by a temperature parameter (T > 1) before applying the softmax function.

• Effect: This 'softens' the output probability distribution, increasing the weights of non-target classes and making the dark knowledge more pronounced.
• Purpose: A softened distribution provides a more informative gradient for the student model to learn from than a peaky, high-confidence distribution.
• Training: The same temperature is applied to the student's logits during distillation, and the standard temperature (T=1) is used for the final inference.

Logits Distillation is the most direct form of knowledge transfer, where the student model is trained to match the raw logits or the temperature-scaled softmax outputs of the teacher. This method directly utilizes Soft Targets as the primary supervisory signal. The training objective typically combines a distillation loss (e.g., Kullback-Leibler Divergence) between the teacher and student distributions with a standard cross-entropy loss using true labels. It forms the foundational approach from which more advanced feature- and attention-based distillation techniques have evolved.

The Teacher-Student Framework is the foundational paradigm for knowledge distillation. It consists of two primary components:

• Teacher Model: A large, pre-trained, and high-performance model (often an ensemble) that provides the supervisory signals (Soft Targets, features, attention maps).
• Student Model: A smaller, more efficient architecture designed for deployment, which learns to replicate the teacher's behavior. The framework can be offline (static, pre-trained teacher) or online (teacher and student co-trained). This structure is essential for understanding the directional flow of knowledge, where Soft Targets are the most common form of transmitted information.

Kullback-Leibler (KL) Divergence Loss is the most commonly used objective function for distilling knowledge from Soft Targets. It quantitatively measures the difference between two probability distributions—in this case, the softened output distribution of the teacher and the student.

• Function: It minimizes the information loss when using the student's distribution to approximate the teacher's.
• Application: In distillation, the loss is calculated on the temperature-scaled softmax outputs. It is often combined with a standard cross-entropy loss against the true labels to ensure the student learns both the dark knowledge and the correct hard targets.