Contrastive Learning

The core mechanism relies on defining semantically similar (positive) and dissimilar (negative) pairs. In self-supervised settings, positives are often generated via data augmentation—applying transformations like cropping, rotation, or color jitter to the same source image. Negatives are typically other distinct samples within the batch. The quality of these pair definitions directly dictates the granularity of the learned representation; poorly constructed negatives can lead to class collision, where a sample semantically similar to the anchor is incorrectly pushed away.

The de facto standard objective, InfoNCE (Noise Contrastive Estimation), operates as a categorical cross-entropy loss over a softmax distribution. It maximizes the mutual information between positive pairs by identifying the correct positive among a set of negative distractors. Key properties include:

• Temperature Parameter (τ): Controls the concentration of the distribution; lower values create harder penalties for difficult negatives, improving feature uniformity.
• Gradient Dynamics: Implicitly performs hard negative mining, where the gradient contribution is proportional to the similarity of a negative sample to the anchor.

Contrastive frameworks typically employ a dual-encoder or Siamese architecture where two branches process augmented views. A critical empirical finding is the use of a non-linear projection head (a small MLP) on top of the encoder backbone. The contrastive loss is computed on the output of this head, but the representation used for downstream tasks is taken from the layer before it. This prevents the loss from discarding information that is useful for discrimination but irrelevant to the specific contrastive pretext task, preserving more generalizable features.

Contrastive learning benefits significantly from a large number of negative samples to approximate the true data distribution. This is achieved through:

• Large Batch Sizes: Frameworks like SimCLR use massive batches (e.g., 4096 or 8192) to treat other in-batch samples as negatives.
• Memory Banks: MoCo maintains a dynamic dictionary of encoded representations from preceding batches, decoupling the dictionary size from the batch size. A momentum encoder updates the dictionary keys smoothly to ensure consistency, preventing stale representations from degrading training.

A well-trained contrastive embedding space exhibits two key geometric properties:

• Alignment: Positive pairs should be mapped to nearby feature vectors, minimizing the distance between augmented views of the same sample.
• Uniformity: The distribution of all feature vectors should preserve maximal information by being roughly uniform on the unit hypersphere, preventing dimensional collapse where all representations converge to a trivial constant vector. The InfoNCE loss implicitly balances these two objectives, acting as a repulsive force between all non-positive samples.

Not all negatives contribute equally to learning. Hard negatives—samples that are difficult to distinguish from the anchor—provide the strongest training signal. Advanced strategies include:

• Debiased Contrastive Loss: Corrects for the sampling bias introduced when negatives are drawn from the same batch, which may accidentally contain positives.
• Hard Negative Sampling: Explicitly selecting the most similar items from the candidate pool to serve as negatives, accelerating convergence by focusing the model on decision boundaries where confusion is highest.

The standard objective function for contrastive learning, derived from Noise Contrastive Estimation. It treats the task as a multi-class classification problem where the goal is to identify the true positive sample among a set of negative distractors.

• Formulated as a categorical cross-entropy over similarity scores.
• A low temperature parameter sharpens the distribution, focusing the model on hard negatives.
• Directly maximizes mutual information between related views.

A metric learning objective that operates on triplets: an anchor, a positive sample, and a negative sample. It enforces a margin between relative distances.

• Minimizes distance between anchor and positive.
• Maximizes distance between anchor and negative by at least a specified margin alpha.
• Sensitive to hard negative mining; random triplets often produce zero loss, stalling training.

The process of selecting negative samples that are difficult to distinguish from positives. Random negatives are often too easy and provide no useful gradient.

• Offline mining: selecting hard negatives from a fixed dataset before training.
• Online mining: selecting the hardest negatives within the current mini-batch.
• Critical for learning fine-grained distinctions in embedding space.

The standard metric for comparing embeddings in contrastive learning. It measures the cosine of the angle between two vectors, ignoring magnitude.

• Computed as the dot product of L2-normalized vectors.
• Range: -1 (opposite) to 1 (identical).
• Normalization ensures all embeddings lie on the unit hypersphere, stabilizing training.

The broader paradigm that contrastive learning fits into. Models learn representations from unlabeled data by solving a pretext task.

• SimCLR: Contrasts augmented views of the same image.
• MoCo: Uses a momentum encoder and a dynamic queue to decouple batch size from dictionary size.
• SimSiam: Demonstrates that simple stop-gradient operations prevent collapse without negative samples.

The technique of creating semantically invariant views of the same input to define positive pairs. The choice of augmentation defines what the model learns to ignore.

• Vision: random cropping, color jittering, Gaussian blur.
• Text: back-translation, word deletion, synonym replacement.
• Sequential: cropping, masking, permuting subsequences.