DeiT (Data-efficient Image Transformer)

DeiT introduces a learnable [class] distillation token that flows through the transformer encoder alongside the patch tokens and the standard class token. This token's purpose is to specifically aggregate the information needed to match the teacher's predictions. The model is trained with a combined loss function:

• A cross-entropy loss between the class token's output and the ground-truth label.
• A Kullback-Leibler (KL) divergence loss between the distillation token's output and the softened predictions from the CNN teacher. This dual-token architecture cleanly separates the two learning objectives.

DeiT employs a specific form of distillation called hard-label distillation. While it uses the teacher's soft predictions (with temperature scaling) to compute the KL divergence loss, the final training objective treats the teacher's predicted class as a true label. This means the student learns to match both the ground truth and the teacher's top-1 prediction. This method proved more effective for ViTs than traditional soft distillation, providing a strong, direct learning signal that complements the standard cross-entropy loss.

Prior to DeiT, high-performing Vision Transformers required pre-training on enormous, often private datasets like JFT-300M (300 million images). DeiT's distillation strategy eliminated this dependency. By learning from a CNN teacher pre-trained on ImageNet-1k (1.2 million images), the DeiT model achieved competitive top-1 accuracy (e.g., 83.1% for DeiT-Base) using only the publicly available ImageNet data. This democratized high-performance vision transformer research and deployment.

DeiT's training recipe heavily utilizes aggressive data augmentation and regularization strategies to prevent overfitting on the mid-sized ImageNet dataset, which is critical when training a high-capacity transformer. Key techniques include:

• RandAugment and MixUp for data augmentation.
• CutMix for combining images.
• Random Erasing.
• Stochastic Depth (DropPath) and Repeated Augmentation. This robust training schedule is as crucial to DeiT's success as the distillation mechanism itself.

DeiT demonstrated that a standard Vision Transformer (ViT) architecture, without convolutional stems or hierarchical designs, was sufficient for top-tier image classification when trained properly. The flagship DeiT-Base model uses the same configuration as ViT-Base:

• Patch size: 16x16 pixels.
• Hidden size: 768.
• Transformer layers: 12.
• Attention heads: 12.
• Parameters: 86 million. This proved the viability of the pure transformer paradigm for vision, provided an effective training strategy like distillation is used.

Knowledge Distillation (KD) is the overarching model compression technique where a compact student model is trained to replicate the behavior of a larger, pre-trained teacher model. The student learns not just from hard ground-truth labels but from the teacher's softened output probabilities, which contain dark knowledge about inter-class relationships. This process enables the creation of models that are faster and require less memory while preserving much of the teacher's accuracy.

• Core Mechanism: Uses a distillation loss (e.g., KL Divergence) to align the student's predictions with the teacher's soft targets.
• DeiT Context: DeiT employs a convolutional neural network (CNN) as a teacher model to distill visual inductive biases into the Vision Transformer student.

A teacher model is a large, high-accuracy, and typically pre-trained neural network whose knowledge is transferred to a smaller student during distillation. In feature-based distillation, the student may learn from the teacher's intermediate attention maps or feature representations. The teacher's role is to provide a rich, informative training signal beyond simple dataset labels.

• DeiT's Choice: DeiT famously uses a RegNetY-16GF CNN as its teacher, proving that a strong CNN can provide effective supervisory signals to a transformer student, eliminating the need for massive, proprietary image datasets like JFT-300M.
• Output: The teacher provides soft targets via temperature-scaled logits.

This distinction is fundamental to distillation's effectiveness.

• Hard Label: A one-hot encoded vector (e.g., [0, 0, 1, 0]) representing the single, ground-truth class from the dataset. It contains no information about similarity between classes.
• Soft Target: A probability distribution (e.g., [0.05, 0.15, 0.75, 0.05]) produced by applying a softmax function (with temperature scaling) to the teacher model's logits. It reveals the teacher's "certainty" and relational understanding—for instance, that a "tabby cat" is somewhat similar to a "lynx" but very different from a "truck." This dark knowledge is what the student model learns to mimic.

The distillation loss is the objective function that trains the student to mimic the teacher. It is typically a weighted combination of two components:

1. Standard Cross-Entropy Loss: Computed between the student's predictions and the ground-truth hard labels. This ensures basic task correctness.
2. Mimicry Loss: Measures the discrepancy between the student's and teacher's soft targets. The most common mimicry loss is the Kullback-Leibler Divergence (KL Divergence) Loss, which quantifies how one probability distribution differs from another.

• DeiT's Formula: L_global = (1 - λ) * L_CE(student, hard_label) + λ * τ² * L_KL(student_soft, teacher_soft) where λ balances the two losses and τ is the temperature parameter.

Attention Transfer is a feature-based distillation method where the student is trained to replicate the attention maps generated by the teacher model's intermediate layers. In Vision Transformers, these maps indicate which spatial regions of an image the model deems most important for making a prediction. By forcing the student to adopt similar attention patterns, it learns the teacher's spatial reasoning and focus.

• Mechanism: A loss function (e.g., Mean Squared Error) minimizes the distance between teacher and student attention maps.
• Relation to DeiT: While DeiT's primary distillation uses output logits, subsequent vision transformer distillation research often incorporates attention transfer to improve student performance further, capturing intermediate representational knowledge.

Data-Free Distillation is a technique to train a student model using only a pre-trained teacher model, without access to the original training data. This is critical for privacy, licensing, or when the original dataset is unavailable. The method typically involves generating synthetic samples that effectively "probe" the teacher's knowledge.

• Common Approaches:
  • Using Batch Normalization statistics stored in the teacher to generate representative inputs.
  • Adversarial generation where a generator creates samples that maximize the student's mimicry loss.
• Contrast with DeiT: DeiT relies on the ImageNet dataset for its distillation. Data-free methods represent a more extreme and challenging scenario for knowledge transfer, highlighting the value of the data-efficient, but not data-free, approach used in DeiT.