Soft Targets

A soft target is a vector of probabilities over all possible classes, not just the correct one. For a three-class problem, a teacher might output [0.85, 0.12, 0.03] instead of the hard label [1, 0, 0]. The non-zero probabilities on incorrect classes encode dark knowledge—the teacher's learned similarity structure. For example, an image of a cat might have a 0.03 probability for 'dog' and 0.001 for 'airplane', revealing that cats are more similar to dogs than to airplanes. This relational information is the core value of soft targets.

A hyperparameter T applied to the softmax function that controls the softness of the output distribution:

• T = 1: Standard softmax, producing sharp, confident distributions
• T > 1: Softens the distribution, amplifying the relative probabilities of incorrect classes
• T → ∞: Produces a uniform distribution where all classes are equally likely

The softened distribution reveals the dark knowledge hidden in the teacher's logits. A typical distillation temperature ranges from 2 to 20, depending on the complexity of the task and the capacity of the student model.

Soft targets provide significantly more information per training example than hard labels. A hard label for a 1000-class problem conveys exactly log₂(1000) ≈ 10 bits of information. In contrast, a soft target distribution conveys the full covariance structure between classes, effectively providing orders of magnitude more supervision. This is why a student model trained on soft targets can often match teacher performance with far fewer examples, and why distillation is sometimes called model compression through rich supervision.

Soft targets are used in the Kullback-Leibler (KL) divergence term of the distillation loss function. The total loss is typically:

codeCopy
L = α * L_CE(student_logits, hard_labels) + (1-α) * T² * L_KL(student_softmax/T, teacher_softmax/T)

• L_CE: Standard cross-entropy with ground truth
• L_KL: KL divergence between temperature-scaled student and teacher distributions
• T² multiplier: Compensates for the gradient scaling effect of temperature
• α: Weighting hyperparameter balancing the two objectives

When the teacher is an ensemble of models, soft targets encode the collective uncertainty and diversity of the ensemble members. The resulting distribution represents a Bayesian model average over the ensemble's predictions. A single student trained on these ensemble-derived soft targets can capture the generalization benefits of the entire ensemble—including reduced overfitting and calibrated uncertainty estimates—without the inference cost of running multiple models. This is one of the most powerful applications of distillation.

The gradient of the cross-entropy loss with respect to the logits for a soft target is proportional to (p_teacher - p_student). This means the student receives a directional correction signal for every class, not just the target class. Even when the student correctly predicts the top class, it continues to learn from the teacher's relative probabilities on all other classes. This dense gradient signal explains why distillation often converges faster and to better minima than training on hard labels alone.