InfoNCE

InfoNCE frames representation learning as a multi-class classification problem. Given an anchor sample, the model must correctly identify the single positive sample from a set of N-1 negative distractors. The loss is computed as:

• The numerator is the exponentiated cosine similarity between the anchor and the positive sample, scaled by a temperature parameter τ
• The denominator sums the exponentiated similarities between the anchor and all samples in the batch (both positive and negatives)
• Minimizing this loss maximizes the probability of correctly classifying the positive pair

This formulation directly estimates the mutual information between representations, providing a lower bound on I(x; c) where x is the input and c is the context.

The temperature parameter τ critically controls the concentration of the similarity distribution and the penalty applied to hard negatives:

• Low τ (< 0.1): Creates a peaked distribution, heavily penalizing hard negatives that are close to the anchor. This enforces fine-grained discrimination but can destabilize training
• High τ (> 0.5): Produces a smoother distribution, treating all negatives more uniformly. This stabilizes early training but may fail to separate subtle distinctions
• Optimal range: Typically between 0.07 and 0.5, tuned as a hyperparameter based on batch size and embedding dimensionality

The temperature effectively scales the logits before the softmax, determining how much the model cares about separating difficult negative samples from the positive.

InfoNCE leverages in-batch negatives for computational efficiency, treating all other samples in the mini-batch as negative examples for each anchor:

• For a batch of size B, each anchor has 1 positive and 2(B-1) negatives (when using symmetric loss with two augmented views)
• This eliminates the need for a separate memory bank or queue, simplifying the architecture
• Larger batch sizes provide more negatives, improving the mutual information bound and representation quality
• The approach introduces a sampling bias: semantically similar samples may be incorrectly treated as negatives, which debiased contrastive loss variants address

This technique is fundamental to frameworks like SimCLR and CLIP, where batch size directly correlates with downstream task performance.

The Normalized Temperature-scaled Cross Entropy Loss (NT-Xent) is the specific InfoNCE variant used in SimCLR with key modifications:

• Embeddings are L2-normalized to unit hypersphere before computing similarity, making cosine similarity equivalent to dot product
• The loss is computed symmetrically: both views of a positive pair serve as the anchor, doubling the training signal
• Uses a large batch size (typically 4096+) to provide sufficient in-batch negatives without a memory bank
• Strong data augmentation (random cropping, color distortion, Gaussian blur) creates the positive pairs from the same source image

NT-Xent demonstrated that simple contrastive frameworks with proper augmentation can match or exceed supervised pre-training on ImageNet.

The gradient of InfoNCE with respect to the anchor embedding reveals its repulsion-attraction mechanics:

• Positive attraction: The gradient pulls the anchor toward the positive sample, weighted by (1 - P(positive|anchor))
• Negative repulsion: Each negative pushes the anchor away, weighted by P(negative_i|anchor), the model's current probability of confusing that negative for the positive
• Hard negatives receive stronger gradients: Negatives with high similarity to the anchor contribute more to the repulsion force, automatically focusing learning on difficult distinctions
• This implicit hard negative mining property makes InfoNCE particularly effective without requiring explicit negative selection strategies

The gradient structure explains why temperature tuning is critical: it directly modulates the probability distribution that weights these forces.

InfoNCE provides a variational lower bound on mutual information between representations, formalized as:

• I(x; c) ≥ log(N) - L_InfoNCE, where N is the number of samples and L is the loss value
• Minimizing the loss directly maximizes the lower bound on mutual information between the anchor and context representations
• This connects contrastive learning to information theory, providing theoretical justification beyond empirical performance
• The bound becomes tighter as N increases, explaining why larger batch sizes or memory banks improve representation quality
• This property distinguishes InfoNCE from simpler losses like triplet loss, which lack this information-theoretic grounding

The mutual information perspective unifies contrastive learning with predictive coding and variational inference frameworks.

The Normalized Temperature-scaled Cross Entropy Loss is the specific variant of InfoNCE popularized by SimCLR. It operates on L2-normalized embeddings and introduces a temperature parameter to control the concentration of the similarity distribution.

• Uses cosine similarity as the scoring function
• Symmetric formulation: computes loss for both (i,j) and (j,i)
• Temperature τ typically set between 0.07 and 0.5
• Lower temperatures create sharper distributions, penalizing hard negatives more aggressively

A training efficiency technique where other samples within the same mini-batch are reused as negative examples for a given positive pair. This eliminates the need for a separate memory bank or explicit negative sampling.

• Negative pool size equals batch_size - 1 per positive
• Larger batches provide more diverse negatives
• Risk of sampling bias: semantically similar items may be incorrectly treated as negatives
• Debiased contrastive loss corrects for this sampling error

The strategic selection of negative samples that are deceptively similar to the anchor but belong to a different class. These samples force the model to learn fine-grained discriminative features rather than relying on superficial differences.

• Can be mined from top-k ANN search results
• Improves embedding quality for near-duplicate detection
• Too many hard negatives can cause training instability
• Often combined with curriculum learning schedules

A critical hyperparameter in InfoNCE and NT-Xent that scales the logits before the softmax operation. It directly controls the concentration of the similarity distribution and the relative penalty applied to hard negatives.

• Low τ (< 0.1): Sharp distribution, heavy penalty on hard negatives, risk of overfitting
• High τ (> 0.5): Smooth distribution, uniform penalty, risk of underfitting
• Optimal values depend on batch size and embedding dimensionality
• Acts as an implicit entropy regularizer

A failure mode where the encoder maps all inputs to a constant or near-identical vector, making the loss trivially zero but destroying all semantic information in the embedding space.

• InfoNCE prevents collapse by explicitly repelling negatives
• Architectures like BYOL and SimSiam avoid collapse without negatives using stop-gradient and momentum encoders
• VICReg and Barlow Twins use covariance regularization instead
• Monitoring embedding variance during training detects early collapse