Self-Distillation

In self-distillation, the teacher and student networks share the same architectural backbone but differ in parameter states. The teacher is typically an exponential moving average (EMA) of the student's weights, updated via the rule θ_t ← mθ_t + (1-m)θ_s. This creates a temporally ensembled model that produces more stable and higher-quality target representations than the rapidly updating student. Unlike traditional knowledge distillation, no external pre-trained model is required—the supervision signal emerges entirely from the model's own evolving representations.

Self-distillation frameworks must actively prevent representation collapse, where the model outputs a constant vector for all inputs. Three primary strategies are employed:

• Sharpening: The teacher's output distribution is sharpened using a low softmax temperature, concentrating probability mass on high-confidence predictions
• Centering: A running mean of teacher outputs is subtracted to prevent dominance by a single dimension
• Stop-Gradient: Gradients are blocked from flowing through the teacher network, breaking symmetry and preventing both branches from converging to identical trivial solutions These mechanisms, central to frameworks like DINO and BYOL, ensure the student learns meaningful features rather than shortcut solutions.

The momentum coefficient m controls how rapidly the teacher network tracks the student's evolving weights. Typical values range from 0.996 to 0.999, creating a slowly evolving teacher that acts as a temporal ensemble of past student states. This slow evolution serves two purposes:

• Provides stable regression targets for the student, reducing training variance
• Accumulates knowledge across training iterations, producing higher-quality representations than any single snapshot During training, m often follows a cosine schedule, starting lower to allow rapid initial alignment and increasing toward 1.0 for stability in later epochs.

Self-distillation methods commonly employ multi-crop augmentation, generating multiple views at different resolutions from a single image:

• Global views (2 large crops, e.g., 224×224): Passed through the teacher to produce high-quality target representations
• Local views (multiple small crops, e.g., 96×96): Passed through the student, which must match the teacher's global-view outputs This local-to-global consistency objective forces the student to learn that a small patch (e.g., a tumor margin) belongs to the same semantic context as the full image, significantly improving feature quality without additional supervision.

The self-distillation objective minimizes the cross-entropy between the student's predictions and the teacher's softened probability distributions. The loss is computed as: L = -Σ P_t(x) log P_s(x) where P_t is the teacher's sharpened output and P_s is the student's prediction. The soft targets carry dark knowledge—information about inter-class similarities encoded in the teacher's probability mass on incorrect classes. For example, in medical imaging, the teacher might assign 5% probability to a benign condition when the true label is malignant, teaching the student about visual similarities between conditions that hard labels would ignore.

Self-distillation excels in medical imaging where labeled data is scarce but unlabeled scans are abundant:

• Chest X-ray pre-training: Models learn anatomical structures and pathological patterns from millions of unlabeled radiographs before fine-tuning on small annotated datasets
• Cross-modality transfer: A model self-distilled on CT data provides strong initialization for MRI tasks, learning modality-agnostic visual features
• Anomaly detection: Self-distilled representations capture normal anatomical variation so precisely that deviations indicating disease are easily identified
• Domain adaptation: Self-distillation on target-domain unlabeled images reduces distribution shift between training and deployment populations without requiring target-domain labels