SimSiam

The core innovation of SimSiam. The gradient flow is blocked on one branch of the Siamese network, treating its output as a fixed target. This asymmetric update prevents the encoder from finding a trivial collapsed solution where all outputs are constant. The stop-gradient operation is implemented by detaching the target branch from the computational graph, forcing the online branch to predict a non-differentiable representation without the target network adapting to make the task easier.

A small multi-layer perceptron appended only to the online branch, transforming its output before comparing it to the stop-gradient target. The predictor introduces an asymmetry that breaks the symmetry of the twin networks. Without it, the stop-gradient alone is insufficient—the model would still collapse. The predictor is typically a 2-layer MLP with a bottleneck hidden dimension, followed by batch normalization and ReLU activation.

SimSiam proves that negative pairs are not necessary for contrastive learning. Unlike SimCLR or MoCo, which require explicit negative samples to prevent collapse, SimSiam achieves stable training using only positive pairs (two augmented views of the same image). This dramatically simplifies the training pipeline by eliminating the need for large batch sizes, memory banks, or careful negative mining strategies.

The paper posits that SimSiam implicitly implements an Expectation-Maximization (EM) optimization procedure. The stop-gradient operation introduces a form of alternating optimization between two sets of variables, preventing the representation from degenerating. This hypothesis challenges the prevailing assumption that momentum encoders (as in BYOL) are required for stability, showing that a simple stop-gradient suffices when combined with a predictor.

Unlike SimCLR, which requires batch sizes of 4096 or larger to provide sufficient in-batch negatives, SimSiam trains effectively with batch sizes as small as 64 or 128. This makes the method accessible on hardware with limited GPU memory and removes the engineering complexity of distributed batch normalization or gradient accumulation across multiple devices.

SimSiam computes a symmetric loss by feeding two augmented views through the network in both directions. View x1 is encoded by the online branch and predicted to match the stop-gradient encoding of view x2, and vice versa. The final loss is the average of both directions. This symmetry ensures the model learns consistent representations regardless of which view serves as the target.

The primary failure mode that SimSiam is designed to prevent without negative pairs or momentum encoders. Collapse occurs when the encoder maps all inputs to a constant vector or a low-dimensional subspace, achieving zero loss trivially but destroying all semantic information.

• Complete collapse: All outputs become identical regardless of input
• Dimensional collapse: Outputs span only a few dimensions of the embedding space
• SimSiam proves that a stop-gradient operation alone breaks the symmetry that causes collapse

Bootstrap Your Own Latent was the first architecture to demonstrate that negative pairs are not strictly necessary for self-supervised learning. BYOL uses a momentum encoder (exponential moving average of the online network) as a target to prevent collapse.

• Online network predicts target network's output via a predictor MLP
• SimSiam directly challenged BYOL's core assumption by removing the momentum encoder entirely
• The key insight: the predictor and stop-gradient are sufficient; momentum is an optimization aid, not a necessity

A slowly evolving copy of the main encoder updated via exponential moving average (EMA): θ_target ← m·θ_target + (1-m)·θ_online, where m is typically 0.999. Used in MoCo and BYOL to maintain consistent representations.

• SimSiam's core contribution is demonstrating that this component is unnecessary for collapse prevention
• Removing the momentum encoder simplifies the architecture to a single network processing two views
• The stop-gradient operation on one branch provides sufficient asymmetry

Two data samples from different semantic instances that contrastive models push apart in embedding space. Traditional methods like SimCLR require large batches (4096+) to provide sufficient negatives.

• SimSiam operates without any negative pairs, avoiding the computational burden of large batch sizes
• This eliminates the need for in-batch negatives or memory banks
• The architecture relies purely on maximizing agreement between positive pairs while the stop-gradient prevents degenerate solutions

A Simple Framework for Contrastive Learning of Visual Representations that relies on large batch sizes and strong data augmentation. SimCLR uses NT-Xent loss with in-batch negatives to pull positive pairs together while pushing all other samples apart.

• Requires batch sizes of 4096+ for competitive performance
• SimSiam achieves comparable results with batch size 256 and no negatives
• Both frameworks share the principle of maximizing invariance to data augmentation, but SimSiam strips away the contrastive mechanism

The critical operation that SimSiam identifies as the essential component preventing collapse. Applied to one branch of the Siamese network, it blocks gradient flow: the target encoder receives no update from the loss on that pathway.

• Implemented as detach() in PyTorch or stop_gradient() in TensorFlow
• Creates an asymmetric update rule that prevents the encoder from finding the trivial collapsed solution
• SimSiam's hypothesis: stop-gradient introduces the necessary asymmetry that momentum encoders and negative pairs were indirectly providing