Contrastive Training for OOD

The fundamental dynamic of contrastive learning operates through positive pairs (augmented views of the same image) and negative pairs (different samples). The loss function minimizes the distance between positive pairs in the embedding space while maximizing the distance between negative pairs. This creates a structured latent space where semantic similarity directly corresponds to geometric proximity, making it trivial to identify inputs that fall outside the learned manifold.

SimCLR (Simple Framework for Contrastive Learning of Visual Representations) established the modern contrastive learning paradigm. Key components include:

• Stochastic data augmentation: Random cropping, color distortion, and Gaussian blur
• Projection head: A small MLP that maps representations to a space where contrastive loss is applied
• NT-Xent loss: Normalized temperature-scaled cross-entropy loss operating on cosine similarities
• Large batch sizes: Typically 4096+ to provide sufficient negative examples without a memory bank

MoCo addresses the need for large negative sample pools without massive batch sizes. It maintains a dynamic dictionary as a queue of encoded representations, decoupling dictionary size from batch size. A momentum encoder—updated via exponential moving average rather than backpropagation—ensures consistent key representations over time. This architecture enables effective training on commodity hardware while maintaining the discriminative power needed for OOD detection.

Contrastively trained models exhibit a distinctive property: in-distribution features have consistently higher L2 norms than OOD features. This occurs because the contrastive objective encourages tight clustering of known classes, while OOD inputs fail to align with any learned prototype direction. Simple thresholding on feature norms often outperforms complex OOD scoring methods, providing a computationally cheap and highly effective detection mechanism.

SupCon extends self-supervised contrastive learning by leveraging label information to construct positive sets. Instead of treating only augmented views as positives, all samples from the same class are pulled together. This creates more compact and well-separated class clusters than cross-entropy training alone. For OOD detection, these tighter clusters produce sharper decision boundaries and more reliable confidence estimates on unknown inputs.

The quality of negative samples critically impacts representation quality. Hard negatives—samples that are difficult to distinguish from the anchor—provide the strongest training signal. Techniques include:

• Debiased contrastive loss: Corrects for sampling bias when negatives share the same class
• Ring loss: Penalizes norm variance to stabilize training
• Adversarial negatives: Synthetically generated challenging examples Effective hard negative mining directly improves the separation between ID clusters and the OOD region.

An extension of self-supervised contrastive learning that leverages label information to pull embeddings of samples from the same class together while pushing apart clusters of samples from different classes. Unlike standard cross-entropy, it operates directly on normalized embeddings, creating a feature space where inter-class margins are maximized. This is critical for OOD detection because it prevents the feature collapse often seen in softmax-trained models, ensuring unknown inputs fall into low-density regions rather than overlapping with known class clusters.

The canonical loss function used in frameworks like SimCLR. It computes the cosine similarity between positive pairs relative to all negative pairs in a mini-batch, scaled by a temperature parameter (τ). The temperature controls the concentration of the distribution; lower values sharpen the penalty on hard negatives. For OOD detection, proper tuning of τ is essential—it directly influences the uniformity and tolerance of the embedding space, determining how tightly in-distribution features cluster and how far OOD features are repelled.

A sampling strategy that prioritizes negative examples which are difficult to distinguish from the anchor sample. In the context of OOD, hard negatives are auxiliary outliers or samples from semantically similar but distinct classes that lie near the decision boundary. By aggressively contrasting against these challenging examples, the model learns a tighter, more precise in-distribution manifold. This prevents the common failure mode where OOD inputs from a proximate domain (e.g., a new dog breed) are incorrectly assigned high confidence.

Two key properties of a well-trained contrastive embedding space. Alignment measures how close positive pairs are in the latent space, while uniformity measures how evenly features are distributed on the unit hypersphere. A perfectly uniform distribution prevents dimensional collapse and ensures that OOD inputs do not inadvertently map to a dense in-distribution cluster. Contrastive training explicitly optimizes for both, making it superior to standard classification for OOD tasks where the goal is to maximize the distance between the known manifold and the rest of the space.