SimCLR

SimCLR applies a stochastic composition of augmentations to generate two correlated views of the same image. The standard pipeline includes:

• Random cropping followed by resize back to original dimensions
• Random color distortions (color jittering and color dropping)
• Random Gaussian blur

Color distortion is particularly critical—without it, the model exploits the color histogram shortcut to solve the pretext task, learning trivial features rather than generalizable representations.

A small non-linear MLP head maps encoder representations to a latent space where contrastive loss is applied. Key design choices:

• The head is a 2-3 layer MLP with a ReLU hidden layer
• Output dimension is typically 128-dimensional
• The head is discarded after pre-training; downstream tasks use the encoder's penultimate layer

This architecture prevents the contrastive loss from discarding information useful for downstream tasks by isolating invariance learning in the projection space.

The Normalized Temperature-scaled Cross Entropy loss operates on L2-normalized embeddings. For a positive pair (i,j) in a batch of N samples:

• All other 2(N-1) augmented samples serve as in-batch negatives
• The temperature parameter τ (typically 0.1) controls concentration—lower values sharpen the distribution, penalizing hard negatives more severely
• Loss is computed symmetrically for both (i,j) and (j,i)

This formulation maximizes mutual information between augmented views while leveraging large batch sizes for diverse negative sampling.

SimCLR's performance scales directly with batch size, using LARS optimizer to maintain stability:

• Batch sizes range from 256 to 8192
• Larger batches provide more in-batch negatives, improving the contrastive signal
• Training on 128 TPU v3 cores enables batch sizes impractical on single-GPU setups
• The LARS optimizer applies layer-wise adaptive learning rates, preventing divergence at extreme batch scales

This design choice eliminates the need for a memory bank by treating the current batch as a sufficiently large negative pool.

SimCLR is encoder-agnostic—it works with any backbone network:

• Original experiments used ResNet-50 with varying widths (1×, 2×, 4×)
• Larger models benefit more from self-supervised pre-training, with ResNet-50 (4×) matching supervised ResNet-50 on ImageNet
• The framework extends naturally to Vision Transformers and other architectures

This flexibility makes SimCLR a general-purpose pre-training recipe rather than a model-specific technique.

After the encoder produces a spatial feature map, global average pooling collapses spatial dimensions into a single vector before the projection head. This design:

• Aggregates features across all spatial locations into a holistic image representation
• Removes spatial structure, forcing the model to capture global semantic content rather than local texture matching
• Contrasts with dense contrastive methods that operate on local patches

The pooling operation ensures the learned representations encode scene-level semantics suitable for classification and detection downstream tasks.

The specific loss function powering SimCLR. Normalized Temperature-scaled Cross Entropy Loss operates on L2-normalized embeddings and uses a temperature parameter to control the concentration of the similarity distribution.

• Computes cosine similarity between all pairs in a batch
• Treats the two augmented views of the same image as the positive pair
• All other 2(N-1) augmented views in the batch serve as in-batch negatives
• A lower temperature value sharpens the distribution, penalizing hard negatives more severely

A training efficiency technique central to SimCLR's design. Instead of maintaining an external memory bank, every other sample in the current mini-batch is reused as a negative example.

• Eliminates the need for a separate momentum encoder or queue
• Requires large batch sizes (4096–8192) to provide sufficient negative diversity
• Simplicity is the key advantage: no extra infrastructure beyond the batch itself
• Trade-off: batch size becomes a critical hyperparameter for representation quality

A critical failure mode where the encoder maps all inputs to a constant or highly similar vector, making the loss trivially zero but destroying all semantic information.

• Complete collapse: all outputs converge to a single point
• Dimensional collapse: individual embedding dimensions become highly correlated
• SimCLR prevents collapse through the temperature parameter and the use of many diverse negatives
• Frameworks like BYOL and SimSiam prevent collapse without any negative pairs at all, using stop-gradient and predictor networks

The augmentation pipeline defines what the model learns to be invariant to. SimCLR demonstrated that composition of augmentations is crucial for learning generalizable representations.

• Random cropping and resizing was identified as the single most important augmentation
• Color distortion (jittering and dropping) prevents the model from cheating via color histograms
• Gaussian blur and random horizontal flips add further robustness
• The positive pair is simply two differently augmented views of the same source image—no labels required

An extension that leverages explicit class labels to generalize the positive pair definition. Instead of only treating augmented views of the same instance as positives, all samples from the same class become positives.

• Pulls together all embeddings belonging to the same category
• Pushes apart embeddings from different categories
• Produces tighter intra-class clusters than standard SimCLR
• Outperforms cross-entropy on ImageNet classification benchmarks when used as a pre-training objective