Data-efficient Image Transformer (DeiT)

DeiT employs a hard-label distillation approach where a convolutional neural network (typically a RegNet or ConvNet) acts as a teacher. The student ViT is trained with a combined loss: the standard cross-entropy loss with ground-truth labels and a distillation loss that matches the teacher's hard predictions. This transfers the inductive biases of convolutions—locality and translation equivariance—directly into the Transformer, compensating for the lack of these priors in self-attention.

• Distillation token: A dedicated learnable token is added to the input sequence alongside the class token
• The distillation token interacts with the teacher's output, while the class token interacts with the ground truth
• At inference, both tokens can be averaged for a performance boost

DeiT relies heavily on aggressive data augmentation and regularization to prevent overfitting on smaller datasets. The recipe combines multiple techniques that are critical for ViT convergence without large-scale pre-training:

• Rand-Augment: A learned augmentation policy that randomly applies transformations like rotation, shear, and color jittering
• Mixup and CutMix: Input-level mixing strategies that blend two images and their labels, forcing the model to learn smoother decision boundaries
• Repeated Augmentation: Each image is transformed multiple times within a batch, increasing effective diversity
• Stochastic Depth: Randomly drops entire Transformer blocks during training with a linearly increasing survival probability

The DeiT training protocol specifies a precise optimization configuration that stabilizes Transformer training on ImageNet-scale data:

• AdamW optimizer with a cosine learning rate decay schedule and linear warmup
• Weight decay applied only to weights, not biases or layer normalization parameters
• Label smoothing with a factor of 0.1 to prevent overconfidence
• Gradient clipping at a maximum norm of 1.0 to prevent exploding gradients
• Training for 300 epochs with a batch size of 1024, significantly longer than typical CNN schedules

DeiT provides a family of models at different capacity points, all using the same training recipe. The base architecture mirrors the standard ViT design with 12 Transformer encoder layers, 12 attention heads, and an embedding dimension of 768 for the base variant.

• DeiT-Tiny: 5M parameters, 72% ImageNet top-1 accuracy
• DeiT-Small: 22M parameters, 79.8% top-1 accuracy
• DeiT-Base: 86M parameters, 81.8% top-1 accuracy
• DeiT-Base distilled: 86M parameters, 83.4% top-1 accuracy when using the distillation token

All variants use a patch size of 16×16 pixels and process 224×224 input images.

A key architectural innovation in DeiT is the introduction of a distillation token that is processed in parallel with the standard class token. This design choice has important implications:

• The distillation token learns to reproduce the teacher's hard label prediction
• The class token learns from the true ground-truth label
• At inference, the outputs of both tokens can be combined via averaging, providing an ensemble-like effect from a single model
• The distillation token can be fine-tuned separately from the class token on downstream tasks
• This dual-token design enables the model to simultaneously capture both the teacher's inductive biases and the true data distribution

Unlike the original ViT which required pre-training on JFT-300M (a proprietary dataset of 300 million images) to achieve competitive ImageNet results, DeiT trains exclusively on ImageNet-1k (1.28 million images). This democratizes Vision Transformer training by:

• Eliminating dependence on massive proprietary datasets
• Reducing total training compute by orders of magnitude
• Enabling reproducible research with publicly available data
• Making ViT architectures accessible to academic labs and smaller organizations
• Demonstrating that architectural inductive biases can be compensated by better training recipes