BYOL

BYOL's core architecture consists of two neural networks: an online network and a target network.

• The online network is the primary model being trained. It comprises an encoder (e.g., ResNet), a projector, and a predictor.
• The target network has an identical architecture but its parameters are an exponential moving average (EMA) of the online network's parameters. It provides stable, slowly evolving regression targets.
• The online network learns by trying to predict the target network's output representation of a differently augmented view of the same input image.

The target network's parameters are not trained via gradient descent. Instead, they are updated via an exponential moving average of the online network's parameters after each training step.

• Update rule: θ_target ← τ * θ_target + (1 - τ) * θ_online, where τ is a momentum coefficient close to 1 (e.g., 0.99).
• This creates a slowly evolving target, preventing a collapse where the online network outputs a constant representation. The asymmetry between the rapidly updated online network and the slowly updated target network is crucial for learning.

A critical, asymmetric component is the predictor head, a small multi-layer perceptron (MLP) attached only to the online network.

• Its role is to predict the target network's projected representation from the online network's projection.
• This asymmetry prevents a trivial solution. If the predictor were removed or placed on both networks, the model could easily converge to a collapsed state where both networks output identical, constant vectors regardless of input.
• The predictor must learn to adapt the online representation to match the target, forcing the upstream encoder to learn meaningful features.

BYOL operates in two distinct representation spaces to facilitate stable learning.

• Projection Space: The output of the projector MLP (present in both networks). This is a lower-dimensional space (e.g., 256-4096 dimensions) where the mean squared error (MSE) loss is applied.
• Representation Space: The output of the encoder (before the projector). This is the high-dimensional feature space used for downstream tasks via linear evaluation or fine-tuning.
• The loss is computed as the normalized L2 distance between the online network's prediction and the target network's projection: L = ||q_θ(z_θ) - z'_ξ||², where vectors are L2-normalized.

Like other self-supervised methods, BYOL relies heavily on a stochastic data augmentation pipeline to create the two "views" of an image.

• A random subset of augmentations is applied independently to create two correlated views, v and v', from a single input image x.
• Standard augmentations include:
  • Random cropping and resizing
  • Color jitter (brightness, contrast, saturation, hue)
  • Gaussian blur
  • Solarization
  • Grayscale conversion
• The online network processes view v; the target network processes view v'. The model learns an invariant representation to these transformations.

BYOL uses a simple mean squared error (MSE) loss between L2-normalized vectors.

• The core loss is asymmetric: L_θ,ξ = ||q_θ(z_θ) - z'_ξ||², where q is the predictor, z is the online projection, and z' is the target projection.
• In practice, the loss is symmetrized. A second pass is computed where the roles of the two augmented views are swapped: view v' goes to the online network and view v goes to the target network, yielding L'_θ,ξ.
• The total loss is L_total = L_θ,ξ + L'_θ,ξ. This ensures the model is invariant to the specific choice of which view is "online" and which is "target."

BYOL's primary advantage is its removal of the need for negative samples. This avoids the collapsing solution problem where all inputs map to the same point, a common failure mode in non-contrastive methods. It simplifies the training dynamic by focusing solely on pulling positive views together, without the computational and algorithmic complexity of managing a large set of negatives. This can lead to more stable training and removes the dependency on large batch sizes or memory banks for effective negative sampling.

BYOL prevents collapse through architectural asymmetry. The online network uses a predictor head (a small MLP) that the target network lacks. This asymmetry stops the system from finding a trivial, constant solution. The predictor must learn to transform the online representation to match the target, forcing the online network to develop richer, more predictive features. This is a key non-contrastive mechanism distinct from methods that rely on negative pairs or explicit regularization like covariance matrices.

The momentum encoder (target network) provides stable, slowly evolving targets for the online network. Its parameters are an exponential moving average of the online network's parameters. This prevents a rapid co-adaptation and collapse between the two networks. The momentum coefficient (τ, typically >0.99) is a critical hyperparameter:

• A high τ provides very stable targets, aiding convergence.
• A low τ makes the target network update faster, which can destabilize training. This mechanism decouples the target generation from the immediate gradient updates, creating a form of consistency regularization.

BYOL is highly dependent on a strong, carefully designed data augmentation pipeline. The method learns invariance to the transformations applied to create the two views. Weak augmentations provide an easy task that may not force the network to learn robust features, while overly aggressive augmentations can make the prediction task impossible. The composition, strength, and stochasticity of augmentations like random cropping, color jitter, Gaussian blur, and solarization are crucial hyperparameters that directly impact final representation quality and must be tuned for the target domain.

While BYOL eliminates negative pairs, it introduces other costs:

• Dual Network Forward Passes: Two full forward passes (online + target) are required per batch, doubling the inference cost during training compared to a single encoder.
• Predictor Head: An additional small MLP must be trained and maintained.
• Batch Normalization Dependencies: Early implementations relied heavily on BatchNorm within the networks to prevent collapse, introducing dependencies on batch statistics. While later work addressed this, it remains a consideration for small-batch or distributed training scenarios. The total cost is often comparable to contrastive methods but with a different profile.

BYOL introduces several sensitive hyperparameters whose interaction must be carefully managed:

• Momentum coefficient (τ): Governs target network update speed.
• Learning rate schedule: Must be coordinated with the momentum updates.
• Predictor architecture and LR: The predictor often uses a separate, higher learning rate.
• Weight decay: Plays a more pronounced role in regularization. Empirically, BYOL can be less forgiving than some contrastive methods if these are set suboptimally. Achieving stable convergence across different datasets and architectures often requires more extensive tuning, making automated hyperparameter optimization valuable for production use.

Contrastive learning is a self-supervised paradigm where a model learns by distinguishing between similar (positive) and dissimilar (negative) data pairs. It explicitly pushes representations of different samples apart while pulling augmented views of the same sample together. This is the dominant approach BYOL was designed to improve upon by eliminating the need for negative samples.

• Core Mechanism: Uses a loss function like InfoNCE that requires comparing each sample against many negatives.
• Key Challenge: Performance heavily depends on the quantity and quality of negative samples, requiring large batch sizes or memory banks.
• Examples: SimCLR and MoCo (Momentum Contrast) are foundational contrastive methods.

Non-contrastive learning is a class of self-supervised methods that learn useful representations without directly comparing negative samples. Instead, they prevent representation collapse through architectural constraints like asymmetry, stop-gradient operations, or regularization terms. BYOL is a seminal non-contrastive method.

• Core Principle: Avoids the need for explicit negative pairs, which can be computationally expensive and sensitive to batch composition.
• Collapse Prevention: Methods use tricks like a predictor head, stop-gradient, and a momentum encoder (as in BYOL) or variance/covariance regularization (as in Barlow Twins and VICReg).
• Advantage: Often enables effective learning with smaller batch sizes compared to contrastive approaches.

A momentum encoder is a slowly evolving, parameterized copy of the main online encoder, updated via an exponential moving average (EMA). It provides stable, consistent target representations for the online network to predict, which is critical for preventing collapse in non-contrastive methods like BYOL and MoCo.

• Update Rule: θ_target ← τ * θ_target + (1 - τ) * θ_online, where τ is a momentum coefficient close to 1 (e.g., 0.99).
• Function: Acts as a "teacher" network, producing targets that change slowly, giving the "student" online network a consistent objective to follow.
• System Benefit: Decouples the target representation generation from the rapidly updated online network, adding stability.

SimSiam is a simple non-contrastive learning architecture that, like BYOL, avoids negative samples and large batches. It uses a siamese network with a predictor head on one side and a stop-gradient operation on the other. Crucially, it demonstrates that a momentum encoder is not strictly necessary for preventing collapse, challenging initial assumptions about BYOL.

• Key Components: A stop-gradient operation on the target branch and a predictor MLP on the online branch.
• Difference from BYOL: SimSiam does not use a momentum encoder; both branches share weights except for the predictor. The stop-gradient operation is sufficient to prevent collapse.
• Insight: Showed that asymmetry (via the predictor) is the critical factor for stability in non-contrastive learning.

Barlow Twins is a non-contrastive method that reduces redundancy between the components of a learned embedding. It achieves this by making the cross-correlation matrix between the embeddings of two distorted views of a batch close to the identity matrix. This directly enforces invariance to distortions while decorrelating the embedding dimensions.

• Loss Function: Minimizes a sum of two terms: one for invariance (diagonal elements → 1) and one for redundancy reduction (off-diagonal elements → 0).
• Comparison to BYOL: Unlike BYOL's asymmetric prediction task, Barlow Twins uses a symmetric, purely regularization-based objective. It requires no predictor, stop-gradient, or momentum encoder.
• Advantage: Conceptually simple and hyperparameter-light, relying primarily on the batch size and the weight of the redundancy term.

Continual self-supervised learning (CSSL) is the problem of training a model on a never-ending, non-stationary stream of unlabeled data. The goal is to learn generalizable representations incrementally without catastrophically forgetting previously learned features. BYOL's architecture presents interesting properties for this setting.

• Core Challenge: Avoiding feature drift—where the model's internal representations change detrimentally as it adapts to new data, harming performance on past tasks.
• BYOL's Relevance: The momentum encoder provides a form of stability, as its slow update may act as a gentle regularizer against rapid forgetting of old features.
• Research Direction: Investigating how non-contrastive methods like BYOL, with their inherent stability mechanisms, can be integrated with experience replay or regularization for effective CSSL.