Contrastive Loss

Contrastive loss operates on paired data—explicitly labeled as similar (positive) or dissimilar (negative). The loss function applies a pull force to positive pairs, minimizing their distance in the embedding space, and a push force to negative pairs, ensuring they exceed a defined margin.

• Positive Pairs: Loss is proportional to the squared Euclidean distance between their embeddings.
• Negative Pairs: Loss is zero if the distance already exceeds the margin; otherwise, it penalizes the model for placing them too close.
• Margin Parameter: A hyperparameter that defines the minimum acceptable distance between dissimilar samples, preventing the model from collapsing all points to a single cluster.

The standard contrastive loss for a pair of samples i and j with label Y (1 for similar, 0 for dissimilar) and distance D is defined as:

L = Y * D² + (1 - Y) * max(0, margin - D)²

• Similar (Y=1): The loss becomes D², directly minimizing the distance.
• Dissimilar (Y=0): The loss becomes max(0, margin - D)², which is non-zero only if the distance is less than the margin.
• This formulation creates an energy-based landscape where similar objects fall into low-energy valleys and dissimilar objects are separated by high-energy barriers.

While both are metric learning losses, they differ in input structure and optimization goals. Contrastive loss uses pairs, while triplet loss uses triplets (anchor, positive, negative).

• Pairwise vs. Relative: Contrastive loss defines absolute similarity thresholds. Triplet loss enforces a relative constraint: the anchor must be closer to the positive than to the negative by a margin.
• Training Stability: Triplet loss requires careful hard negative mining to avoid trivial solutions. Contrastive loss can be more stable with well-defined pairs but may converge more slowly.
• Use Case: Contrastive loss is preferred for tasks with explicit similarity labels (e.g., duplicate detection), while triplet loss excels in ranking and retrieval where relative order matters.

In modern dense retrieval training, explicit negative pairs are often scarce. In-batch negatives is a technique that reuses other positive pairs within the same mini-batch as negative examples for a given query.

• Efficiency: A batch of N query-passage pairs yields N positives and N² - N negatives without additional labeling.
• Implementation: The loss is computed over a similarity matrix of size N x N, where the diagonal represents positive pairs and off-diagonal entries are negatives.
• Gradient Amplification: This dramatically increases the number of negative examples per step, accelerating convergence and improving the discriminative power of the encoder.

Random negatives are often too easy, providing little learning signal. Hard negatives—samples that are superficially similar to the query but irrelevant—force the model to learn fine-grained distinctions.

• Mining Strategies: Hard negatives can be retrieved using a sparse model like BM25 or a previous checkpoint of the dense retriever itself.
• Loss Integration: These hard negatives are added to the contrastive loss calculation, often with a higher weight, to specifically penalize high-similarity false positives.
• Effect: Training with hard negatives significantly improves Recall@K, as the model learns to separate semantically related but distinct concepts.

Contrastive loss is the training objective for distilling a powerful cross-encoder into a fast bi-encoder. The cross-encoder scores query-passage pairs with full attention, generating high-quality relevance labels.

• Teacher-Student Setup: The cross-encoder (teacher) scores a large set of passages for each query. The top-scoring passage is the positive, and lower-scoring ones serve as weighted negatives.
• Margin Adaptation: The margin in the contrastive loss can be dynamically scaled based on the teacher's score difference, creating a smoother training signal.
• Outcome: The resulting bi-encoder achieves retrieval accuracy approaching the teacher's while maintaining the speed required for searching billion-scale vector indexes.

A training efficiency technique that reuses other positive pairs within a mini-batch as negative examples. Instead of explicitly mining negatives, every other passage in the batch serves as a negative for a given query.

• Dramatically increases the number of negatives per step without extra computation
• Works best with large batch sizes (often 128–4096)
• Forms the backbone of most DPR training pipelines
• Can introduce false negatives if the batch contains semantically similar but unlabeled pairs

Training examples that are superficially similar to the query but factually irrelevant, forcing the model to learn fine-grained distinctions.

• Typically sourced from top BM25 or ANN retrieval results that don't match the ground truth
• Prevent the model from relying on lexical overlap as a shortcut
• Mining strategies include static top-K retrieval and dynamic in-epoch re-mining
• Critical for performance on challenging benchmarks like Natural Questions

A loss formulation that operates on (anchor, positive, negative) triplets, requiring the anchor-positive distance to be smaller than the anchor-negative distance by a specified margin.

• Defined as: max(0, d(a,p) − d(a,n) + margin)
• The margin hyperparameter controls how aggressively dissimilar pairs are pushed apart
• Contrastive loss can be viewed as a special case with margin = 0 and specific pairing
• Often used with online triplet mining to select the hardest negatives within a batch

The Normalized Temperature-scaled Cross Entropy loss, popularized by SimCLR, extends contrastive learning to handle multiple positive and negative pairs simultaneously.

• Applies L2 normalization to embeddings before computing cosine similarities
• Uses a temperature parameter (τ) to control the concentration of the distribution
• Lower temperatures sharpen the distribution, focusing learning on the hardest negatives
• Forms the theoretical foundation for many modern dense retrieval training objectives

A slowly updating copy of the query encoder used to maintain a consistent representation space when training with large negative queues.

• Updated via exponential moving average: θ_k ← m·θ_k + (1−m)·θ_q
• Typical momentum coefficient m ranges from 0.999 to 0.9999
• Prevents representation collapse when negatives come from different training steps
• Key component of MoCo (Momentum Contrast) and related self-supervised frameworks

A knowledge transfer process where a computationally expensive cross-encoder teaches a fast bi-encoder to improve its retrieval accuracy.

• The cross-encoder scores query-passage pairs with full self-attention, providing soft labels
• The bi-encoder is trained to mimic these scores using KL divergence or MSE loss
• Yields bi-encoders that approach cross-encoder quality at a fraction of the inference cost
• Enables deployment of high-accuracy semantic search at scale