SimSiam

SimSiam uses a siamese network, a structure with two identical subnetworks (encoders) that process two different augmented views of the same input image. The encoders share weights. The core objective is to make the representations of these two views agree with each other, learning an invariant representation to the applied augmentations. This architecture is the foundation for defining the positive pair without external data.

A critical component preventing representational collapse is the predictor head, a small multi-layer perceptron (MLP) attached to one side of the siamese network. This creates architectural asymmetry: one view passes through the encoder and predictor, while the other passes only through the encoder. The predictor learns to output a projection that matches the target projection from the other view. This asymmetry prevents a trivial, collapsed solution where both networks output constant vectors.

The stop-gradient (stopgrad) operation is the mechanism that makes SimSiam's asymmetric learning stable. In the loss calculation, the gradient is not backpropagated through the target branch's encoder. Formally, the loss is: L = -sim(p(z₁), sg(z₂)), where p() is the predictor, z are projections, and sg() is stopgrad. This treats one side's output as a constant target, akin to a slowly moving teacher in distillation, which empirically prevents collapse without negative samples.

Unlike contrastive methods like SimCLR or MoCo, SimSiam is a non-contrastive method. It does not require negative pairs—explicit examples of dissimilar data used to push representations apart. This simplifies the training objective and removes the need for large batch sizes or memory banks, which are typically used to provide a rich set of negatives. Learning is driven purely by maximizing similarity between positive views, enabled by the stopgrad and predictor.

The training loss is symmetrized to ensure both augmented views contribute equally to learning. For a batch of images, two random augmentations are created for each. The loss is computed twice: first with view A predicting view B, then with view B predicting view A. The final loss is the average of these two terms: L = ½ [L(A→B) + L(B→A)]. This ensures the encoder learns a consistent representation regardless of which view serves as the predictor input.

SimSiam's simplicity makes it a compelling candidate for continual self-supervised learning. Its lack of a dynamic dictionary or negative queue avoids complications with non-stationary data streams. However, applying it continually introduces challenges:

• Feature Drift: The encoder's representation space can drift over time.
• Predictor Forgetting: The predictor head may become misaligned with the evolving encoder.
• Augmentation Consistency: Maintaining a consistent augmentation strategy across tasks is critical for stable learning. Research explores integrating SimSiam with experience replay or regularization to mitigate catastrophic forgetting.

Contrastive learning is a self-supervised technique that trains a model to produce similar embeddings for semantically related data points (positive pairs) and dissimilar embeddings for unrelated points (negative pairs). It relies on a loss function, like InfoNCE, that explicitly compares samples.

• Core Mechanism: Pulls positive pairs together and pushes negative pairs apart in the embedding space.
• Key Challenge: Requires careful construction of negative pairs, often needing large batch sizes or memory banks.
• Example: SimCLR uses two augmented views of the same image as a positive pair and all other images in the batch as negatives.

SimSiam is notable for achieving strong performance without this explicit negative sampling mechanism.

BYOL is a pioneering non-contrastive self-supervised method that, like SimSiam, eliminates the need for negative pairs. It uses two neural networks: an online network and a target network.

• Architecture: The online network tries to predict the target network's output of a differently augmented view of the same image.
• Key Innovation: The target network's parameters are an exponential moving average (EMA) of the online network, providing stable, slowly evolving targets.
• Difference from SimSiam: While both avoid negatives, BYOL uses this momentum encoder (EMA) and does not use a stop-gradient operation on one branch. SimSiam simplifies this by using a stop-gradient and a predictor head on a single symmetric network.

The stop-gradient operation is a critical, simple trick in SimSiam that prevents a representational collapse where all outputs become identical. It is applied to one branch of the siamese network during the loss calculation.

• Function: It treats the output from one branch (e.g., from the view 2 encoder) as a constant target. Gradients are not backpropagated through this branch for that part of the loss.
• Effect: It creates an asymmetric gradient flow, forcing the other branch (with the predictor) to learn meaningful representations to match the static target.
• Analogy: Acts like a simplified, detached momentum encoder without the need for an EMA update. This is a key factor enabling SimSiam's simplicity and stability.

Non-contrastive learning is a category of self-supervised methods that learn useful representations without directly comparing negative samples. SimSiam, BYOL, Barlow Twins, and VICReg belong to this family.

• Core Principle: Avoid the computational and theoretical complexity of managing negative pairs. Instead, they prevent collapse through architectural asymmetry (e.g., predictor heads, stop-gradient) and regularization terms (e.g., redundancy reduction).
• Common Techniques:
  • Predictor Heads: Small MLPs (as in SimSiam/BYOL) that break symmetry.
  • Regularization Losses: Terms that enforce variance, covariance, or decorrelation in the embedding dimensions.
• Advantage: Often more memory-efficient and can work with smaller batch sizes than contrastive methods.

A Siamese network is a neural architecture consisting of two or more identical subnetworks (they share weights) that process different inputs. It's foundational to SimSiam and many contrastive methods.

• Purpose: To process two or more inputs (e.g., different augmented views of an image) and produce comparable output vectors.
• In SimSiam: The network has two identical encoder branches (f). One branch's output passes through an additional predictor head (h), and its output is compared to the stop-gradient output of the other branch.
• Key Feature: Weight sharing ensures the same feature extraction logic is applied to both inputs, which is essential for learning invariant representations.

Representational collapse is a failure mode in self-supervised learning where a model learns a trivial solution, such as mapping all inputs to the same constant vector, which minimizes the loss without learning useful features.

• Cause in Siamese Nets: Without a prevention mechanism, symmetric networks can quickly converge to this trivial solution.
• Prevention Methods:
  • Contrastive Loss: Uses negative pairs to discourage collapse (e.g., SimCLR).
  • Architectural Asymmetry: Introduces a component that breaks symmetry, like SimSiam's predictor head and stop-gradient.
  • Regularization: Methods like Barlow Twins add a term to decorrelate embedding dimensions.

SimSiam's primary contribution is demonstrating that simple asymmetry (predictor + stop-gradient) is sufficient to avoid collapse, removing the need for negatives, large batches, or momentum encoders.