DINO

DINO employs a Siamese architecture where the teacher network's weights are an exponential moving average (EMA) of the student's weights, not shared. The student learns by predicting the teacher's output on different augmented views of the same image. This momentum update rule ensures the teacher provides stable, high-quality target representations that evolve smoothly over time, avoiding the instability of using a copy of the student as the target.

DINO solves the representation collapse problem without using negative pairs, contrastive loss, or large batches. It applies two complementary operations to the teacher's output:

• Centering: Subtracts a running mean of teacher outputs, preventing one dimension from dominating.
• Sharpening: Applies a low temperature softmax to the teacher's output, encouraging peaky, confident pseudo-labels. This asymmetric centering-sharpening dynamic naturally balances the learning signal and prevents trivial solutions.

A remarkable emergent property of DINO-trained Vision Transformers (ViTs) is that the self-attention maps of the final block automatically segment salient foreground objects from the background. Without any pixel-level supervision, the [CLS] token's attention heads learn to focus on semantically coherent object boundaries. This makes DINO features directly usable for unsupervised object discovery, semantic segmentation, and copy-paste detection tasks.

DINO introduces a multi-crop augmentation strategy to balance computational cost and representation quality:

• Global views: Two standard-resolution crops (e.g., 224x224) covering large portions of the image.
• Local views: Several low-resolution crops (e.g., 96x96) capturing fine-grained details. All crops are passed through the student, but only global views go through the teacher. This encourages the student to learn local-to-global correspondences, significantly boosting performance on downstream tasks without the quadratic cost of processing all pairs at full resolution.

DINO evaluates learned representations using a k-Nearest Neighbors (k-NN) classifier on frozen features, rather than linear probing or fine-tuning. This protocol directly measures the quality of the embedding space without introducing additional learned parameters. A k-NN classifier with k=10 or k=20 is applied to the frozen backbone features, and top-1 accuracy is reported. This evaluation is more sensitive to the geometric structure of the representation space and correlates strongly with transfer learning performance.

DINOv2 extends the original framework by training on a carefully curated 142-million-image dataset (LVD-142M) with automatic data filtering pipelines. Key improvements include:

• KoLeo regularizer: Encourages uniform feature coverage across the embedding space.
• Sinkhorn-Knopp centering: Replaces the original centering with batch-wise optimal transport normalization.
• FlashAttention: Enables efficient training of larger ViT models (up to ViT-g/14). DINOv2 features serve as a universal visual backbone, achieving strong performance across depth estimation, semantic segmentation, and image retrieval without fine-tuning.

A slowly evolving copy of the main encoder network updated via exponential moving average (EMA). In DINO, the teacher network is a momentum encoder that provides stable, high-quality target representations for the student to match. The update rule follows: θ_t ← mθ_t + (1-m)θ_s, where m typically starts at 0.996 and increases during training. This slow progression prevents the teacher from collapsing and ensures consistent pseudo-labels across epochs.

Two complementary operations that prevent representation collapse in DINO's self-distillation framework:

• Sharpening: Applies a low temperature (τ_t < τ_s) to the teacher's softmax output, making the target distribution more peaked and confident. This forces the student to learn crisp, discriminative features.
• Centering: Subtracts a running mean from the teacher's logits to prevent one dimension from dominating. The center c is updated via EMA: c ← βc + (1-β)g, where g is the batch mean of teacher outputs.

Together, sharpening encourages specificity while centering ensures diversity across the batch.

The central failure mode that DINO's design explicitly prevents. Collapse occurs when the encoder maps all inputs to an identical or trivially similar output vector, making the loss function meaningless. DINO combats collapse through three mechanisms:

• Centering prevents dominance by a single dimension
• Sharpening forces discriminative predictions
• Momentum encoder provides stable, slowly evolving targets

Without these safeguards, the student-teacher framework would quickly degenerate into a constant-output trivial solution.