Bootstrap Your Own Latent (BYOL)

BYOL employs an asymmetric online-target network pair that learns from two augmented views of the same input. The online network consists of an encoder, a projector, and a predictor, while the target network has only an encoder and projector. The target network's weights are not updated via backpropagation but through an exponential moving average (EMA) of the online network's parameters. This architectural asymmetry is the key mechanism that prevents representation collapse without requiring negative examples.

• Online network: encoder → projector → predictor
• Target network: encoder → projector (no predictor)
• Target parameters: ξ ← τξ + (1-τ)θ (where τ is the EMA decay rate)

The stop-gradient operation is the critical component that prevents BYOL from collapsing to a trivial solution. When computing the loss, gradients flow only through the online network; the target network receives no gradient updates. This breaks the symmetry that would otherwise allow both networks to output constant vectors. The stop-gradient forces the online network to learn meaningful representations by predicting a slowly evolving target, effectively turning the problem into a form of self-distillation where the target acts as a stable teacher.

• Blocks gradient flow to the target network
• Prevents trivial collapsed solutions
• Enables learning without negative pairs

The predictor is a small multi-layer perceptron (MLP) attached exclusively to the online network. Its role is to map the online network's projection to the target network's projection space. The predictor introduces additional asymmetry between the two branches, which is essential for stable training. Without the predictor, the loss would be minimized by making the encoder output constant. The predictor is discarded after pre-training; only the online encoder is retained for downstream tasks like few-shot modulation recognition.

• Typically a 2-layer MLP with batch normalization
• Applied only to the online branch
• Discarded before fine-tuning on labeled RF data

The target network parameters are updated using an exponential moving average (EMA) of the online network's weights, controlled by a decay rate τ (typically starting at 0.996 and increasing to 1.0 during training). This slow evolution ensures the target produces consistent, high-quality representations across training steps. The momentum encoder acts as a temporal ensemble, smoothing out the noise from individual gradient updates and providing a stable regression target for the online network to predict.

• Update rule: θ_target ← τ·θ_target + (1-τ)·θ_online
• τ follows a cosine schedule from 0.996 to 1.0
• Provides stable, slowly evolving target representations

BYOL relies on stochastic data augmentations to create two correlated views of the same input. For radio frequency applications, domain-specific augmentations are applied to raw IQ samples. Effective RF augmentations include additive white Gaussian noise (AWGN), random phase rotation, frequency offset simulation, time cropping, and amplitude scaling. The model learns to produce similar representations despite these perturbations, forcing it to capture the underlying signal structure rather than superficial channel artifacts.

• Phase rotation: simulates carrier offset
• AWGN injection: builds noise robustness
• Time cropping: enforces temporal invariance
• Frequency shift: simulates Doppler effects

Unlike contrastive methods such as SimCLR or MoCo, BYOL does not use negative pairs to prevent collapse. Instead, the combination of the predictor, stop-gradient, and EMA updates creates a learning dynamic where the online network must constantly adapt to match a moving target. The predictor introduces sufficient asymmetry that the trivial solution of outputting a constant vector is not a stable fixed point of the optimization. Empirical results show BYOL achieves state-of-the-art linear evaluation accuracy on ImageNet without any negative examples.

• No negative pairs required
• Asymmetry prevents trivial solutions
• Matches or exceeds contrastive method performance

A slowly evolving copy of the online network, updated via exponential moving average (EMA) rather than backpropagation. In BYOL, the momentum encoder serves as the target network, producing stable and consistent representations for the online network to predict. The update rule follows: θ_target ← τθ_target + (1−τ)θ_online, where τ is typically 0.99-0.999. This slow evolution prevents rapid parameter changes that could destabilize the target and cause representation collapse.

A critical architectural mechanism that blocks gradient flow into the target network during backpropagation. The online network learns to predict the target's output, but the target never receives gradients from this prediction loss. This asymmetry prevents the trivial collapsed solution where both networks output identical constant vectors. The stop-gradient is the key innovation that allows BYOL to eliminate negative pairs entirely while avoiding collapse.

A knowledge distillation paradigm where the student and teacher share identical architectures. In BYOL, the online network (student) is trained to predict the target network's (teacher) representation of an augmented view. Unlike traditional distillation, there is no pre-trained teacher — the teacher is built from the student's own past weights via EMA. This bootstrapping process iteratively refines representations without external supervision.

A failure mode where the encoder produces a constant or non-informative output for all inputs, rendering learned representations useless. BYOL prevents collapse through:

• Stop-gradient asymmetry: prevents both networks from co-adapting to a trivial solution
• Batch normalization: introduces implicit contrast through batch statistics
• Predictor network: adds a learned transformation between online and target representations Without these mechanisms, non-contrastive methods rapidly degenerate.

A contrastive learning framework that maximizes agreement between differently augmented views of the same sample while repelling all other samples in the batch. Unlike BYOL, SimCLR requires large batch sizes (typically 4096+) and explicit negative pairs to prevent collapse. It uses an InfoNCE loss with a temperature parameter to control concentration. The projection head maps representations to a space where contrastive loss is applied, and is discarded before downstream tasks.

A self-supervised method that avoids both negative pairs and momentum encoders by optimizing the cross-correlation matrix between twin network embeddings. The objective drives the matrix toward the identity:

• Invariance term: diagonal elements forced to 1.0 (embeddings invariant to augmentations)
• Redundancy reduction term: off-diagonal elements forced to 0.0 (vector components decorrelated) This elegantly prevents informational collapse while learning useful representations.