Self-Distillation

A sequential self-distillation process where a student model with an identical architecture to the teacher is trained on the teacher's soft targets. The student becomes the teacher for the next generation. This born-again sequence often yields performance improvements over the original model without any architectural changes, as the soft targets provide a richer, denoised training signal that reveals inter-class similarities.

The model leverages its own dark knowledge—the relative probabilities assigned to incorrect classes in its softmax output. By using a high temperature parameter during softmax scaling, the model surfaces subtle similarities between classes (e.g., how '3' resembles '8' in digit recognition). This self-generated structural information acts as a powerful regularizer, preventing overfitting to one-hot labels.

Self-distillation functions as an iterative denoising mechanism. In each cycle, the model's predictions on the training set become the new target distribution for the next cycle. This process progressively smooths out label noise and corrects inconsistencies, effectively allowing the model to bootstrap itself toward a more calibrated and confident set of predictions without external supervision.

Unlike standard knowledge distillation, which requires a separate, often larger teacher model or an ensemble of models, self-distillation operates within a single model lineage. This eliminates the computational overhead of training and maintaining a complex teacher. The model acts as both the source and recipient of knowledge, making it a highly efficient compression and regularization technique for resource-constrained environments.

A dynamic scheduling strategy where the temperature used to soften the self-generated targets is gradually reduced across distillation generations. Early cycles use high temperatures to focus on broad class relationships, while later cycles use lower temperatures to refine decision boundaries. This annealing process guides the model from coarse structural learning to fine-grained discrimination.

A key outcome of self-distillation is markedly better model calibration. By training on its own softened probability distributions, the model learns to produce confidence scores that more accurately reflect the true likelihood of correctness. This reduces expected calibration error (ECE), making the model's predictive uncertainty more reliable for high-stakes downstream decision systems.

A specific self-distillation architecture where a student model with an identical architecture to the teacher is trained on the teacher's soft targets. Unlike standard distillation, the student has the same capacity as the teacher, yet paradoxically achieves superior generalization performance. The process can be iterated for multiple generations, with each born-again network outperforming its predecessor. This demonstrates that the benefit of distillation is not merely compression but a form of implicit regularization that reshapes the loss landscape.

The implicit information about inter-class similarities encoded in the relative probabilities of incorrect classes within a model's softmax output. When a model assigns probabilities like [0.9, 0.08, 0.02] instead of [1, 0, 0], the distribution over the incorrect classes reveals the model's learned similarity structure—for instance, that class 2 is more similar to class 1 than class 3 is. Self-distillation leverages this dark knowledge by using these softened probabilities as training targets, transferring richer supervisory signals than hard labels alone.

A hyperparameter T applied to the softmax function that controls the softness of the output probability distribution. Higher temperatures (T > 1) produce softer distributions that reveal more dark knowledge by amplifying the relative probabilities of incorrect classes. During self-distillation, the same model generates soft targets at a high temperature and is then trained to match these softened distributions, effectively learning from its own uncertainty estimates and internal similarity metrics.

The process of compressing the collective knowledge of a diverse ensemble of models into a single student. Self-distillation can be viewed as a special case where the ensemble is formed by the same model at different training checkpoints or with different stochastic configurations. By averaging predictions across these snapshots, the model captures ensemble-level robustness without the computational cost of maintaining multiple models, effectively transferring the benefits of model averaging into a single inference path.

A composite objective function that combines the Kullback-Leibler divergence between the teacher's and student's soft targets with the standard cross-entropy loss against ground-truth labels. In self-distillation, both the teacher and student are the same model at different stages, and the loss encourages the model to maintain consistency with its own prior predictions while still fitting the data. This dual objective acts as a regularizer, preventing overfitting and improving calibration.

The alignment between a model's predicted probabilities and the empirical likelihood of correctness. Self-distillation has been shown to significantly improve expected calibration error (ECE) by softening overconfident predictions. When a model trains on its own softened outputs, it learns to produce more conservative probability estimates that better reflect true uncertainty, making it particularly valuable for high-stakes applications where reliable confidence scores are critical for decision-making.