BYOL (Bootstrap Your Own Latent)

Unlike natural images, medical scans require domain-specific augmentation to preserve diagnostic integrity. BYOL for radiology employs transformations that respect anatomical constraints:

• Random elastic deformations to simulate soft tissue variation
• Intensity jittering mimicking MRI bias fields
• Cutout simulation replicating metallic artifacts
• Affine transformations (rotation, scaling) within physiological ranges These augmentations create diverse views while ensuring the underlying pathology remains recognizable.

The projection head in BYOL maps backbone features to a space where the prediction loss operates. For medical imaging, the architecture typically uses:

• Backbone: ResNet-50 or Vision Transformer (ViT)
• Projection MLP: 4096 → 4096 → 256 dimensions with BatchNorm
• Prediction MLP: 256 → 4096 → 256 dimensions After pre-training, the projection head is discarded, and the frozen backbone is evaluated via the linear evaluation protocol on downstream classification tasks like pneumonia detection or malignancy scoring.

A self-supervised paradigm that learns representations by pulling positive pairs together and pushing negative pairs apart in embedding space. Unlike BYOL, contrastive methods explicitly require negative samples to prevent representation collapse. Key examples include SimCLR, which uses large batch sizes for in-batch negatives, and MoCo, which maintains a dynamic queue-based dictionary. The core objective is instance discrimination—treating each image as its own class. BYOL demonstrated that competitive representations can be learned without any negative pairs, eliminating the engineering burden of negative sampling strategies and large batch requirements.

A slowly evolving copy of the online network, updated via an exponential moving average (EMA) of the online weights: θ_target ← τθ_target + (1-τ)θ_online where τ is typically 0.996–0.999. This temporal ensembling produces stable, consistent target representations that serve as regression targets for the online network. The momentum encoder is critical to BYOL's stability—it prevents the online network from chasing a rapidly moving target and implicitly introduces an asymmetry that, combined with the stop-gradient operation, prevents representation collapse without requiring negative pairs or explicit regularization terms.

A contrastive learning framework that maximizes agreement between differently augmented views of the same image using the NT-Xent (Normalized Temperature-scaled Cross Entropy) loss. SimCLR relies on large batch sizes (typically 4096+) to provide sufficient negative samples within each mini-batch. Key architectural components include a projection head (MLP) that maps representations before applying contrastive loss and strong data augmentation. While SimCLR achieves excellent performance, its dependence on large batch sizes creates computational bottlenecks that BYOL avoids entirely by operating without negative pairs.

A self-distillation framework closely related to BYOL where a student network matches the output of a momentum-updated teacher. DINO adds sharpening and centering operations to prevent collapse: sharpening via a low-temperature softmax concentrates probability mass, while centering subtracts a running mean to prevent dominance by any single dimension. Unlike BYOL's L2 prediction objective, DINO uses cross-entropy between softmax outputs. Both methods belong to the non-contrastive joint embedding family and achieve strong performance on Vision Transformers, with DINO demonstrating that self-supervised ViTs naturally learn semantic segmentation capabilities in their attention maps.

A critical failure mode in self-supervised learning where the encoder outputs a constant or trivially non-informative vector for all inputs, achieving zero loss without learning meaningful features. Contrastive methods prevent collapse through negative pairs that repel dissimilar samples. BYOL prevents collapse through its asymmetric architecture: the stop-gradient operation on the target branch combined with the predictor network on the online branch creates an implicit contrastive effect without explicit negatives. The exact mechanism remains theoretically debated—some hypothesize that BatchNorm in the predictor implicitly introduces contrastive statistics, while others argue the EMA teacher acts as a dynamic mean teacher that stabilizes training.

BYOL employs a two-stage head architecture beyond the backbone encoder. The projection head (typically a 2-3 layer MLP with BatchNorm and ReLU) maps representations to a space where the loss is applied. The predictor (an additional MLP applied only to the online branch) introduces further asymmetry between the online and target networks. Key design choices:

• Projection head output dimension: 256 (standard for ResNet-50)
• Predictor: Bottleneck architecture with hidden dimension smaller than output
• BatchNorm in predictor: Empirically critical for preventing collapse
• Both heads are discarded after pre-training; only the backbone encoder is used for downstream tasks