Contrastive Learning

The core engine of contrastive learning relies on defining positive pairs—two views of the same underlying concept. In computer vision, this is typically achieved through data augmentation: applying random crops, color jitter, or Gaussian blur to the same source image generates two distinct inputs that must map to nearby vectors. In NLP, back-translation or span corruption creates semantically equivalent text pairs. The model learns to be invariant to these transformations, forcing it to capture high-level semantic features rather than superficial pixel or token patterns. The choice of augmentation strategy directly determines which invariances the final embedding space will exhibit.

Equally critical is the selection of negative samples—data points the model must push away. In-batch negatives, popularized by SimCLR, treat all other samples in a mini-batch as negatives, creating an efficient implicit comparison without a memory bank. Hard negative mining deliberately selects negatives that are close to the anchor in the current embedding space, providing a stronger training signal. MoCo (Momentum Contrast) uses a dynamic queue and a momentum encoder to maintain a large, consistent dictionary of negatives, decoupling the batch size from the number of negative samples and enabling more stable training.

The standard training objective is InfoNCE (Noise Contrastive Estimation), a categorical cross-entropy loss that treats the positive pair as the correct classification among a set of distractors. Mathematically, it computes the similarity between the anchor and the positive, then normalizes it against the sum of similarities to all negatives. This formulation is directly connected to maximizing the mutual information between different views of the same data. A temperature hyperparameter τ controls the concentration of the distribution: lower temperatures sharpen the penalty on hard negatives, while higher temperatures smooth the gradient signal across all samples.

A critical failure mode is dimensional collapse, where the encoder maps all inputs to a trivial constant vector, achieving perfect alignment but zero representational utility. Architectures combat this through several mechanisms: - Stop-gradient operations in Siamese networks like SimSiam prevent collapsing solutions without negative pairs - Batch normalization implicitly introduces contrast between samples - Whitening (as in Barlow Twins) explicitly decorrelates embedding dimensions - Asymmetric architectures (predictor networks) break symmetry to avoid shortcut solutions. Understanding these anti-collapse measures is essential for debugging training failures.

An extension of the self-supervised paradigm, Supervised Contrastive Loss leverages label information to pull together not just augmented views of the same sample, but all samples belonging to the same class. This creates a two-stage training pipeline: first, a contrastive objective learns a representation where class clusters are well-separated; second, a linear classifier is trained on frozen representations. This approach consistently outperforms standard cross-entropy training on robustness benchmarks and transfer learning tasks, as the embedding space learns to capture intra-class variance rather than just decision boundaries.

The quality of a contrastively learned embedding space can be decomposed into two measurable properties: alignment and uniformity. Alignment measures how close positive pairs are in the embedding space—perfect alignment means all augmentations of a sample map to identical vectors. Uniformity measures how evenly distributed the overall set of embeddings is on the unit hypersphere—perfect uniformity means vectors are spread uniformly, preserving maximal information. These metrics, proposed by Wang & Isola (2020), provide a theoretical framework for understanding why contrastive learning works and offer direct diagnostic tools for evaluating learned representations without downstream tasks.

The fundamental distance metric used in contrastive learning to measure semantic closeness. It calculates the cosine of the angle between two non-zero vectors, effectively ignoring differences in magnitude to focus purely on orientation in the embedding space.

• Range: -1 (opposite) to 1 (identical)
• Used in the InfoNCE loss function
• Preferred over Euclidean distance in high-dimensional spaces

The critical mechanism that teaches the model what is dissimilar. Instead of comparing a positive pair against every other data point in the dataset (which is computationally prohibitive), the model randomly selects a small batch of negative examples to push away.

• In-batch negatives: other items in the mini-batch
• Hard negative mining: selecting the most confusing dissimilar items
• Balance is key: too few negatives cause dimensional collapse

Information Noise-Contrastive Estimation is the dominant loss function in modern contrastive learning frameworks like SimCLR and CLIP. It treats the task as a categorical classification problem where the model must identify the true positive pair among a set of negative distractors.

• Based on a softmax cross-entropy formulation
• Temperature parameter controls concentration
• Directly maximizes mutual information between views

The engine that generates positive pairs without human labels. By applying stochastic transformations to a single data point, the model learns to be invariant to irrelevant changes while preserving semantic identity.

• Vision: random cropping, color jitter, blurring
• Language: back-translation, word deletion, span masking
• Graphs: node dropping, edge perturbation
• The choice of augmentation defines the invariance prior

A landmark framework by Google that simplified contrastive learning into a clean, high-performance pipeline. It relies on a large batch size (up to 8192) to provide sufficient negative samples, a non-linear projection head, and a strong composition of data augmentations.

• Proved that contrastive methods could surpass supervised pre-training
• Architecture: Encoder → Projection Head → Contrastive Loss
• Discards the projection head after training for downstream tasks

A strategy to improve representation quality by focusing the model's attention on the most difficult negative examples—those that are semantically close to the anchor but should be distinct. This prevents the model from solving the task with easy negatives and forces it to learn finer-grained features.

• Can be mixed with random negatives for stability
• Essential for metric learning and fine-grained retrieval
• Reduces the risk of gradient starvation