DeiT (Data-efficient Image Transformer)

DeiT's most significant innovation is the introduction of a dedicated distillation token. This is a learnable vector prepended to the input sequence alongside the standard class token and patch tokens. During training, it interacts with other tokens via self-attention and is trained via a distillation loss to mimic the output of a pre-trained teacher model (e.g., a CNN like RegNet). This allows the transformer to directly learn from the teacher's predictions as an additional supervisory signal, effectively combining hard labels with soft targets.

DeiT employs a hard-label distillation strategy. Unlike traditional knowledge distillation that uses softened output probabilities (soft targets), DeiT's student model is trained to match the teacher's predicted hard label (the argmax class). The total loss is a weighted sum:

• Cross-Entropy Loss with ground-truth labels.
• Distillation Loss (also cross-entropy) with the teacher's hard labels. This approach simplifies the training pipeline and proves highly effective, allowing the transformer student to learn inductive biases from a CNN teacher trained on the same dataset.

DeiT was designed for data efficiency. It demonstrates that Vision Transformers can be trained effectively on a single node with 4 GPUs in just 3 days using only the ImageNet-1k dataset (1.3 million images). This was a breakthrough, as prior Vision Transformers (ViT) required large-scale pre-training on private datasets like JFT-300M (300 million images). DeiT achieves this through its distillation framework and strong data augmentation (RandAugment, MixUp, CutMix, Random Erasing) and regularization strategies.

DeiT uses a standard Transformer encoder architecture. Key optimizations include:

• Relative Position Embeddings: Instead of absolute 1D position embeddings, DeiT uses learned relative position biases added to attention scores, which improves generalization to higher-resolution images at inference.
• Optimizer & Scheduling: Trained with AdamW optimizer and a cosine learning rate decay schedule.
• Stochastic Depth: Employs stochastic depth (drop layers) for regularization, with a higher drop rate for deeper layers.
• No Convolutions: The architecture remains purely based on self-attention, with no convolutional inductive biases built into the model itself.

The paper introduced three main variants, balancing model size and accuracy:

• DeiT-Ti (Tiny): 5M parameters, 72.2% top-1 accuracy on ImageNet.
• DeiT-S (Small): 22M parameters, 79.8% top-1 accuracy.
• DeiT-B (Base): 86M parameters, 81.8% top-1 accuracy. Crucially, DeiT-S outperformed the similarly sized ViT-B trained on ImageNet-only by over 4 percentage points, validating the distillation approach. The models also showed strong performance on transfer learning tasks.

DeiT's distillation differs from classic methods in important ways:

• Token vs. Output Distillation: Knowledge is transferred through a dedicated token's interactions, not just by matching final outputs.
• Hard vs. Soft Labels: Uses the teacher's hard class predictions, not softened probability distributions.
• Joint Training: The distillation signal is present from the beginning of training, not applied to a pre-trained student.
• CNN Teacher: Leverages a convolutional teacher (e.g., RegNet) to impart spatial inductive biases that are not native to the transformer architecture, effectively "teaching" it how to see.

The teacher-student framework is the foundational architecture for knowledge distillation. It consists of a pre-trained, often cumbersome teacher model that provides supervisory signals to a trainable student model.

• Teacher Role: Provides softened labels (logits) or intermediate feature maps as training targets. In DeiT, this is a convolutional neural network (CNN) like RegNet.
• Student Role: The model being compressed (e.g., the Vision Transformer in DeiT) that learns to replicate the teacher's predictions.
• Training Dynamics: The student's loss function is a weighted combination of the standard cross-entropy loss with ground truth and a distillation loss (e.g., KL Divergence) that measures discrepancy from the teacher.

The distillation token is a novel, learnable embedding introduced by DeiT that enables native transformer-based distillation. It is prepended to the input sequence alongside the class token.

• Function: Interacts with the image patches through self-attention and is trained specifically to reproduce the output of the teacher model.
• Architectural Innovation: Allows the Vision Transformer to distill knowledge from a CNN teacher during pre-training, without requiring a separate distillation stage or modifying the transformer's core architecture.
• Output: At the final layer, the class token predicts the true label, while the distillation token's output is used to compute the distillation loss against the teacher's prediction.

This distinction is central to how knowledge is transferred. A hard label is a one-hot vector (e.g., [0, 0, 1, 0]), while a soft target is a probability distribution (e.g., [0.05, 0.1, 0.8, 0.05]).

• Hard Label: Contains minimal information, only indicating the correct class.
• Soft Target: Generated by applying a softmax function to the teacher's logits. Contains dark knowledge—the relative probabilities for incorrect classes, revealing similarities (e.g., that a 'tabby cat' is more similar to a 'tiger' than to a 'truck').
• Temperature Scaling: A temperature parameter (T) is used to soften the distribution further (higher T = softer), making the relative differences between classes more pronounced and easier for the student to learn.

Distillation loss is the specialized objective function that quantifies the difference between the teacher and student models' outputs or features, guiding the student's learning.

• Primary Form: For logit distillation, it is typically the Kullback-Leibler (KL) Divergence between the softened outputs of the teacher and student.
• Combined Loss: In practice, the total loss is a weighted sum: L_total = α * L_CE(hard_labels, student) + β * L_KL(teacher_soft_targets, student_soft_targets).
• Variants: Can also be applied to intermediate features (feature distillation), attention maps (attention transfer), or relational structures between samples (relational knowledge distillation).

Online distillation is a training paradigm where the teacher and student models are co-trained simultaneously from scratch, rather than using a static, pre-trained teacher.

• Contrast with DeiT: DeiT uses a static teacher (a pre-trained CNN). Online distillation methods update the teacher's weights alongside the student's during training.
• Benefits: Can be more flexible and avoid the need for a large, pre-existing teacher model. The teacher can adapt and improve as the student learns.
• Example: In Deep Mutual Learning, multiple peer student models teach each other in a collaborative, online fashion without a predefined superior teacher.