Self-Distillation

Born-Again Networks (BANs) are the seminal self-distillation technique where a student model with an identical architecture to the teacher is trained to outperform its own teacher. The process is iterative: the student from generation n becomes the teacher for generation n+1. Key characteristics include:

• Architectural Identity: Teacher and student share the same model blueprint.
• Progressive Refinement: Each generation distills knowledge from the previous, often leading to monotonic performance improvements.
• Loss Function: Typically uses a weighted combination of Kullback-Leibler divergence from the teacher's softened logits and standard cross-entropy with ground-truth labels.

This variant treats different depths within the same network as teacher-student pairs. Knowledge is distilled from deeper, more expressive layers to guide the training of shallower layers.

• Mechanism: Intermediate feature maps or classifier logits from deep layers generate supervisory signals for earlier layers.
• Objective: Helps mitigate vanishing gradients and improves feature discriminativeness in shallow layers.
• Use Case: Particularly effective in very deep networks (e.g., ResNet-1000) where early layers can benefit from the refined representations learned later in training.

The model generates its own soft targets during training, which are then used as a regularization signal alongside hard labels.

• Process: For a given input, the model produces a prediction, applies temperature scaling to create a softened probability distribution, and uses this distribution as an auxiliary training target.
• Effect: Encourages the model to learn the dark knowledge of its own predictive uncertainties and inter-class relationships.
• Advantage: Functions as a form of label smoothing that is adaptive and specific to the model's current state, often improving calibration and generalization.

Used in networks with early exits or internal classifiers, this method distills knowledge from the final, most accurate exit point to earlier, less accurate exits.

• Architecture: A single backbone network has multiple classifier heads attached at intermediate layers.
• Training: The final exit's predictions (logits or features) supervise the training of earlier exit classifiers via distillation loss.
• Deployment Benefit: Enables adaptive inference where easy samples can exit early with high confidence, reducing average latency, while hard samples pass through the full network.

Knowledge transfer occurs concurrently with primary training, without a separate, static teacher phase. The model continuously learns from its own evolving state.

• Dynamic Teacher: The teacher is the model from the previous training step or a moving average of weights (an Exponential Moving Average model).
• Efficiency: Eliminates the need for a separate pre-training and distillation pipeline.
• Common Technique: Often implemented via auxiliary branches with their own classifiers; the ensemble of all branches' predictions provides a richer self-supervisory signal to each individual branch.

Applies principles from contrastive learning to self-distillation. The model learns by maximizing agreement between differently augmented views of the same data sample as processed by the same network or its variants.

• Mechanism: Two views of an input are passed through the network (or twin networks). A contrastive loss pulls their representations together while pushing them apart from representations of other inputs.
• Knowledge Form: Transfers relational knowledge about the feature space structure.
• Benefit: Learns more robust and transferable representations by leveraging instance discrimination as a self-supervised pretext task within the distillation framework.

This application distills knowledge from deeper layers to shallower layers within the same model. The deeper, more abstract features guide the learning of earlier layers, often implemented via auxiliary classifiers or direct feature matching losses. Use cases include:

• Training Stability: Provides additional, rich supervisory signals to intermediate layers.
• Improved Feature Learning: Encourages shallower layers to learn more discriminative representations early in training.
• Architectural Flexibility: Enables the creation of more efficient sub-networks (e.g., early exits) from a single trained model.

In this paradigm, the model distills knowledge to itself during a single training run, without a separate pre-training phase. Common implementations use multiple classifier heads or branches that teach each other. Benefits include:

• Co-training Branches: Different model components (e.g., classifier heads) act as peer teachers, exchanging knowledge via distillation loss.
• No Pre-trained Teacher: Eliminates the need for a separate, costly teacher training stage.
• Regularization Effect: The mutual teaching acts as a strong regularizer, reducing overfitting and improving generalization on the final single-head model.

Self-distillation is used to enhance model robustness against adversarial attacks and label noise. The softened outputs from the model's own predictions provide a more robust training signal than hard labels. Applications focus on:

• Adversarial Defense: Models trained with self-distillation often show increased resilience to input perturbations and adversarial examples.
• Noisy Label Tolerance: The soft labels generated by the model can smooth out the impact of incorrect hard labels in the dataset.
• Calibration Improvement: Leads to better-calibrated models whose predicted confidence more accurately reflects true correctness likelihood.

Self-distillation trains a single network to have accurate early-exit classifiers at intermediate layers. The final layer's knowledge is distilled back to these early classifiers, enabling adaptive inference. This is critical for:

• Dynamic Computation: Easy samples can exit early, saving inference time and compute.
• On-Device Efficiency: Enables latency/accuracy trade-offs on resource-constrained edge devices.
• Unified Training: All exits are trained jointly with self-distillation, ensuring the early classifiers learn from the model's own final, most accurate representations.

The foundational model compression technique where a smaller, more efficient student model is trained to mimic the behavior and outputs of a larger, more powerful teacher model. The core goal is to transfer the teacher's learned knowledge, including its generalization capabilities and dark knowledge (inter-class relationships), into a deployable, compact form. This is typically achieved using a distillation loss, such as Kullback-Leibler divergence, to align the student's outputs with the teacher's softened probability distributions.

The canonical two-stage architectural pattern for knowledge distillation.

• Teacher: A pre-trained, often cumbersome model (e.g., an ensemble or very deep network) that acts as the source of knowledge.
• Student: A smaller, more efficient model (different or similar architecture) that is trained to replicate the teacher. The framework defines the directional flow of information, which in self-distillation is collapsed, as a single model alternates between the teacher and student roles.

A seminal self-distillation technique where a student model with the identical architecture as its teacher is trained to outperform it. The process is iterative:

1. Train a model (Teacher 1).
2. Use Teacher 1 to distill knowledge into a new model of the same size (Student 2).
3. Student 2 becomes the teacher for the next generation. This cycle demonstrates that knowledge transfer can yield a born-again network superior to the original, highlighting the regularization benefits of distillation.

A training paradigm where the teacher and student models are co-trained simultaneously from scratch, rather than using a static, pre-trained teacher. This creates a dynamic, collaborative learning process. In some online setups, like mutual learning, multiple peer students teach each other. Self-distillation can be viewed as an online process where a model's own deeper layers or later training stages provide guidance to its shallower layers or earlier states.

Methods that transfer knowledge from intermediate model representations, not just final outputs.

• Feature Distillation: The student is trained to match the intermediate feature representations or activations from specific layers of the teacher (e.g., via hint learning).
• Attention Distillation: Specific to transformers, where the student replicates the self-attention patterns or matrices of the teacher's layers. These methods provide richer, structural guidance than logit matching alone.