BYOL

BYOL employs two interacting neural networks: an online network and a target network. The online network is trained to predict the target network's representation of the same image under a different augmented view. The target network is not updated via backpropagation; instead, its weights are an exponential moving average (EMA) of the online network's weights, providing a stable regression target.

A critical architectural component that prevents representation collapse. The online network appends a shallow Multi-Layer Perceptron (MLP) called the predictor to its encoder output. The loss is computed between the predictor's output and the target network's projection. This asymmetric design—where the target network lacks a predictor—ensures the learning task is not trivially solved by the identity function.

Unlike SimCLR or MoCo, BYOL does not require negative pairs. It discards the contrastive paradigm entirely. The model learns by minimizing the L2 distance between the online network's prediction and the target network's projection of a positive pair. This eliminates the engineering complexity of hard negative mining and large batch sizes, making training more robust to batch composition.

The target network receives no gradient updates. A stop-gradient operation is applied to the target branch, preventing the loss from being minimized by simply making both networks output identical vectors. This forces the online network to learn meaningful representations by predicting a slowly evolving target, a mechanism later generalized in architectures like SimSiam.

BYOL uses a simple mean squared error (L2 loss) between the L2-normalized prediction vector and the L2-normalized target projection vector. This is a regression objective, not a classification or contrastive one. The loss is symmetrized by feeding each augmented view through both the online and target networks, doubling the training signal per image.

BYOL matches or exceeds the linear evaluation accuracy of supervised pre-training on ImageNet. With a ResNet-50 encoder, BYOL achieves 74.3% top-1 accuracy (linear eval), surpassing SimCLR and MoCo v2. When scaled to larger ResNet-200 architectures, it reaches 79.6%, demonstrating that negative pairs are not a necessity for state-of-the-art self-supervised learning.

A slowly evolving copy of the main encoder network updated via exponential moving average (EMA). In BYOL, the target network acts as a momentum encoder, providing stable regression targets for the online network. Unlike MoCo, BYOL does not use a memory bank of negative samples; the momentum encoder's stability alone, combined with the predictor MLP, prevents representation collapse.

Simple Siamese networks that explore the hypothesis that stop-gradient operations are the critical component for preventing collapse, rather than momentum encoders or negative pairs. SimSiam directly predicts the output of a shared-weight encoder without a momentum target, demonstrating that the predictor MLP and stop-gradient on one branch are sufficient to avoid trivial solutions.

A failure mode in contrastive and self-supervised learning where the encoder maps all inputs to a constant or highly similar output vector, trivializing the loss function. BYOL avoids collapse through an asymmetric architecture: the online network has a predictor MLP that the target network lacks, and the target network is updated via exponential moving average rather than backpropagation.

Information Noise-Contrastive Estimation, a loss function based on categorical cross-entropy that identifies a positive pair among a set of negative samples. BYOL fundamentally differs from InfoNCE-based methods like SimCLR by eliminating negative pairs entirely, relying instead on a bootstrapping mechanism where the online network predicts the target network's representation.

Self-DIstillation with NO labels, a framework where a student network learns to match the output of a momentum teacher network. Like BYOL, DINO uses a teacher-student architecture without negative pairs, but adds sharpening and centering operations on the teacher outputs to prevent collapse, making it particularly effective for Vision Transformers.

A redundancy-reduction objective that learns representations by making the cross-correlation matrix of two distorted views close to the identity matrix. Unlike BYOL's predictive approach, Barlow Twins prevents collapse by explicitly decorrelating vector components, pushing off-diagonal elements of the correlation matrix toward zero while keeping diagonal elements near one.