Stop-Gradient Operation

Demonstrated that stop-gradient alone—without momentum encoder, negative pairs, or large batches—is sufficient to prevent collapse. SimSiam uses identical architectures for both branches.

• Core finding: The stop-gradient operation introduces an implicit Expectation-Maximization optimization dynamic.
• Simplicity: No EMA update; both encoders share weights, but gradients are blocked on one side.
• Predictor MLP: A crucial component on the online branch that breaks symmetry when combined with stop-gradient.

Adapted the stop-gradient operation for Vision Transformers by applying it to the momentum encoder path. MoCo v3 found that freezing the patch projection layer in the momentum encoder was critical for stability.

• Instability discovery: Without proper stop-gradient handling, ViT training exhibits sudden, catastrophic accuracy dips.
• Solution: Stop-gradient on the momentum encoder combined with a frozen random patch projection layer.
• Contrastive backbone: Unlike BYOL, MoCo v3 retains a queue-based negative sample dictionary.

Combines DINO-style self-distillation with masked image modeling, applying stop-gradient to the teacher network that generates patch-level pseudo-labels for masked tokens.

• Dual objective: Online network predicts both global [CLS] token and masked patch tokens from the stop-gradient teacher.
• Online tokenizer: The teacher acts as a learned tokenizer, replacing dVAE-based tokenization used in BEiT.
• Medical relevance: iBOT-style pre-training on medical images learns both global diagnostic features and local anatomical structures.

The primary failure mode that the stop-gradient operation is designed to prevent. In joint embedding architectures, the encoder may converge to a trivial solution where it outputs a constant vector for all inputs, achieving zero loss but learning no useful features. Stop-gradient breaks the symmetry that allows this shortcut.

• Complete Collapse: Output is a single constant regardless of input.
• Dimensional Collapse: Vectors occupy a low-dimensional subspace, losing information across other dimensions.
• Prevention: Asymmetric architectures (stop-gradient) or explicit regularization (variance, covariance) are required.

The seminal non-contrastive framework that popularized the stop-gradient operation. BYOL trains an online network to predict the representations of a target network, where the target network is updated via an exponential moving average of the online weights. The stop-gradient prevents the target branch from receiving gradients, ensuring it serves as a stable regression target.

• Online Network: Receives gradients, includes a predictor head.
• Target Network: Stop-gradient applied, updated via EMA.
• Key Insight: The predictor learns to compensate for the moving average, preventing collapse without negative pairs.

A slowly evolving copy of the primary encoder, updated via an exponential moving average (EMA) of the primary network's weights. The momentum encoder is the typical recipient of the stop-gradient operation, as it must remain stable to provide consistent target representations.

• Update Rule: θ_target ← m * θ_target + (1 - m) * θ_online
• Typical Momentum: m = 0.99 to 0.999
• Purpose: Decouples the target from noisy online updates, smoothing the learning signal.

A minimalist non-contrastive framework that demonstrates stop-gradient is the essential ingredient for preventing collapse, even without a momentum encoder. SimSiam uses identical encoder weights for both branches but applies stop-gradient to one branch and adds a prediction MLP to the other.

• Architecture: Siamese encoders (shared weights) + predictor head.
• Mechanism: Stop-gradient on one branch + predictor on the other.
• Finding: Momentum encoder is helpful but not necessary; stop-gradient is the critical component.

A self-distillation framework where a student network is trained to match the output of a teacher network built with a momentum encoder. Stop-gradient is applied to the teacher branch. DINO adds sharpening and centering operations to further prevent collapse, making it particularly effective for Vision Transformers.

• Teacher: Stop-gradient + EMA update + centering.
• Student: Receives all gradients, uses sharpened softmax.
• Emergent Property: Attention maps naturally segment objects without supervision.

An alternative to stop-gradient that prevents collapse through explicit regularization rather than architectural asymmetry. VICReg applies three loss terms: invariance (MSE between views), variance (hinge loss to maintain variance above a threshold), and covariance (decorrelating embedding dimensions).

• No Stop-Gradient Required: Both branches receive gradients.
• Variance Loss: Prevents dimensional collapse by enforcing a minimum standard deviation per dimension.
• Covariance Loss: Minimizes off-diagonal elements of the covariance matrix.