Self-Supervised Contrastive Learning

The learning objective is defined by a pretext task that generates pseudo-labels from the data itself. Common strategies include:

• Instance Discrimination: Treating each data point as its own class, with augmented views as positive pairs
• Temporal Co-occurrence: Using frames close in time as positives in video or sequential data
• Cross-Modal Alignment: Pairing text with corresponding images, as in CLIP
• Spatial Patching: Predicting the relative position of image patches The quality of learned representations depends heavily on the pretext task's ability to capture meaningful semantic invariances.

A critical component that defines the invariance properties the encoder must learn. Augmentations create multiple views of the same instance while preserving semantic identity:

• Visual: Random cropping, color jittering, Gaussian blur, horizontal flipping, solarization
• Textual: Back-translation, synonym replacement, span masking, word deletion
• Audio: Pitch shifting, time stretching, background noise injection The augmentation strategy must be carefully tuned—too weak and the model learns trivial shortcuts; too strong and semantic content is destroyed.

The core objective pulls positive pairs together while pushing negatives apart. The dominant formulation is InfoNCE (Information Noise-Contrastive Estimation), which frames the task as identifying the true positive among a set of distractors:

• Uses categorical cross-entropy over similarity scores
• The temperature parameter τ controls the concentration of the distribution
• Lower τ values increase the penalty on hard negatives—samples deceptively similar to the anchor
• Variants include NT-Xent (with L2 normalization) and Supervised Contrastive Loss (leveraging class labels for multiple positives)

The selection and quantity of negative samples directly impact representation quality and computational efficiency:

• In-Batch Negatives: Other samples in the mini-batch serve as negatives, enabling efficient training without a memory bank. Requires large batch sizes (e.g., 4096+ in SimCLR)
• Memory Bank: A queue of stored representations from recent batches, used in MoCo to decouple batch size from negative count
• Hard Negative Mining: Strategically selecting negatives that are difficult to distinguish from positives, forcing finer-grained feature learning
• Debiased Contrastive Loss: Corrects for the sampling bias when unlabeled negatives accidentally share the same latent class as the anchor

Representation collapse—where the encoder maps all inputs to a constant vector—is the primary failure mode. Prevention strategies include:

• Negative Pairs: Explicitly repelling dissimilar samples (SimCLR, MoCo)
• Momentum Encoder: A slowly evolving target network updated via exponential moving average (BYOL, MoCo)
• Stop-Gradient: Preventing gradient flow through one branch of a Siamese network (SimSiam)
• Redundancy Reduction: Decorrelating embedding dimensions via cross-correlation matrix regularization (Barlow Twins)
• Variance Regularization: Explicitly penalizing low variance in embedding dimensions (VICReg)

The standard protocol for assessing learned representations without fine-tuning the encoder:

• Freeze the encoder weights after self-supervised pre-training
• Train only a linear classifier on top of the frozen representations
• Performance on downstream tasks (e.g., ImageNet classification) measures representation quality
• This isolates the quality of the learned features from the capacity of the classifier head
• Semi-supervised evaluation tests performance when only a fraction of labels are available
• Transfer learning benchmarks assess generalization across different domains and tasks

A Simple Framework for Contrastive Learning of Visual Representations. SimCLR relies on large batch sizes and aggressive data augmentation to define positive pairs. The architecture processes two augmented views of the same image through a shared encoder and projection head, using NT-Xent loss to maximize agreement. It eliminates the need for a memory bank by using in-batch negatives, treating all other samples in the mini-batch as negative examples.

Momentum Contrast builds a dynamic dictionary with a queue and a slowly progressing momentum encoder. Unlike SimCLR, MoCo decouples the dictionary size from the batch size, enabling contrastive learning on smaller GPU hardware. The momentum encoder is updated via exponential moving average, ensuring consistent key representations over time. The queue stores the most recent K encoded samples, discarding the oldest to maintain freshness.

Bootstrap Your Own Latent trains without negative pairs. An online network learns to predict the output of a target momentum network from a different augmented view. A predictor MLP is attached only to the online branch, creating an asymmetry that prevents representation collapse. The target network is updated via exponential moving average of the online parameters, making the training self-stabilizing without requiring contrastive negatives.

A redundancy-reduction objective that avoids collapse without negative samples. Two distorted views of a batch are fed through a shared encoder. The loss function makes the cross-correlation matrix of the embeddings close to the identity matrix. This forces the embedding components to be invariant to augmentations (diagonal terms) while decorrelating different vector components (off-diagonal terms), preventing informational redundancy.

Self-DIstillation with NO labels applies a teacher-student framework to Vision Transformers. The student learns to match the output of a momentum teacher using sharpened softmax and centering to prevent collapse. The teacher is updated via exponential moving average. DINO produces attention maps that automatically segment objects, demonstrating that self-supervised ViTs learn semantically meaningful features without any supervision.

Contrastive Language-Image Pre-training learns a joint embedding space from 400 million image-text pairs. A dual-encoder architecture processes images and text independently, trained with a symmetric InfoNCE loss across modalities. The resulting embeddings enable zero-shot transfer: classifying images by comparing them to textual descriptions of categories without any task-specific fine-tuning.