Soft Targets

A hard target is a one-hot encoded vector (e.g., [0, 0, 1, 0]) assigning all probability to a single class. A soft target is a probability distribution (e.g., [0.05, 0.15, 0.70, 0.10]) across all classes. This distribution encodes the teacher's relative confidence and class similarities, known as dark knowledge. For instance, an image of a 'husky' might yield high probability for 'dog' but also small, non-zero probabilities for related classes like 'wolf' or 'malamute', which a one-hot label would discard.

Temperature scaling is the core operation that generates soft targets. A temperature parameter (τ) is applied to the teacher's logits (pre-softmax activations) within the softmax function:

softmax(z_i, τ) = exp(z_i / τ) / Σ_j exp(z_j / τ)

• τ = 1: Standard softmax, producing the model's native confidence distribution.
• τ > 1: 'Softens' the distribution, increasing the entropy and making probabilities more uniform. This amplifies the relative differences between non-true classes, making the dark knowledge more accessible for the student to learn from.
• τ → ∞: All classes approach uniform probability (1/N).
• τ < 1: Sharpens the distribution, making it more peaky, approaching a one-hot encoding.

Soft targets provide a denser training signal by preserving inter-class relationships and the teacher's uncertainty. This richness offers several benefits:

• Regularization: The smoother label distribution acts as a regularizer, preventing the student from overfitting to potentially noisy hard labels.
• Gradient Variance Reduction: Provides more stable, lower-variance gradients compared to the sparse, high-variance signal from hard targets.
• Transfer of Generalization: The student learns the teacher's internal representation of similarity between concepts, often leading to better generalization on unseen data than training on hard labels alone.

Training a student model with soft targets typically uses the Kullback-Leibler (KL) Divergence loss, which measures how one probability distribution diverges from a second, reference distribution. The student's softened outputs (using the same τ as the teacher) are matched to the teacher's soft targets.

L_KD = τ^2 * KL( P_teacher(τ) || P_student(τ) )

The τ^2 term is used to ensure the magnitude of the gradient remains roughly constant when changing the temperature. This is often combined with a standard cross-entropy loss with true hard labels in a weighted sum: L_total = α * L_CE + β * L_KD.

In continual learning, soft targets are a key tool for knowledge retention. When learning a new task (Task B), the model's predictions on data from the old task (Task A) are used as soft targets. By applying a distillation loss between the current model's outputs on old data and its past outputs (or a saved copy of the old model's outputs), the model is penalized for changing its behavior on previous tasks. This technique, central to algorithms like Learning without Forgetting (LwF), helps anchor the model's parameters to preserve old knowledge while adapting to new data.

Soft targets are one form of knowledge transferred from teacher to student. Other forms include:

• Logit Distillation: Directly matching the teacher's logits (pre-softmax activations), often using Mean Squared Error loss. This is closely related to soft target distillation but operates on the unscaled logits.
• Feature/Activation Distillation: Matching intermediate representations (e.g., hidden states or feature maps from convolutional layers). Techniques like FitNets use this to guide the student's internal layers.
• Attention Distillation: Matching the attention maps of transformer-based models.

Soft target distillation is often the most straightforward and effective method for transferring the teacher's final predictive knowledge.

Knowledge distillation is a model compression and transfer learning technique where a smaller, more efficient student model is trained to mimic the behavior of a larger, more complex teacher model. The student learns not from hard labels but from the teacher's softened output probabilities (soft targets) or intermediate feature representations, capturing the teacher's "dark knowledge." This process enables the creation of models that are faster, smaller, and more suitable for deployment while retaining much of the original model's performance.

Dark knowledge refers to the rich, non-obvious information contained within the softened probability distribution (logits) of a trained neural network. Unlike a one-hot label that only indicates the correct class, a softened distribution reveals the teacher model's relative confidence across all classes—for example, that a picture of a car is more similar to a truck than to a cat. This relative similarity information between classes is the key training signal transferred via soft targets, providing a more informative gradient for the student model than hard labels alone.

Temperature scaling is the technique used to generate soft targets. A temperature parameter (T) is applied to the softmax function of the teacher model's logits: softmax(logits / T).

• When T = 1, it's the standard softmax.
• When T > 1, the probability distribution is softened, making all classes more uniformly probable and amplifying the dark knowledge in the smaller probabilities.
• This softened distribution is what the student model learns to match, typically using a KL Divergence loss. After training, the student uses a standard softmax (T=1) for inference.

Catastrophic forgetting (or catastrophic interference) is the tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new data. This is the core problem that techniques using soft targets aim to solve in continual learning. When a model learns Task B, the weights optimized for Task A are overwritten, erasing the old knowledge. Methods like Learning without Forgetting (LwF) use the model's own soft targets from previous tasks as a regularization signal to anchor the parameters, preventing drastic drift and preserving stability.

Learning without Forgetting is a seminal continual learning algorithm that directly employs knowledge distillation to mitigate catastrophic forgetting. When learning a new task:

1. The model's predictions on new task data are recorded.
2. The model's predictions on data from the old task(s) are recorded before updating weights. These act as soft targets.
3. The total loss is a combination of standard cross-entropy loss for the new task and a distillation loss (e.g., KL Divergence) that penalizes the model for changing its outputs on the old tasks. This allows the model to integrate new knowledge while consolidating old knowledge without needing to store raw past data.

The teacher-student framework is the foundational architecture for knowledge distillation. It defines the roles of two models:

• The Teacher: A large, accurate, often pre-trained model (e.g., BERT, ResNet-50) that provides supervisory signals.
• The Student: A smaller, more efficient model (e.g., DistilBERT, MobileNet) that learns from the teacher. The framework specifies what knowledge is transferred (logits, features, attention) and how (the loss function). In continual learning, a single model can act as its own teacher from a previous time step, creating a self-distillation loop to preserve knowledge.