SimCLR

A critical component that generates two correlated views from a single input image. SimCLR demonstrated that composition of augmentations—specifically random cropping followed by color distortion—is essential for learning generalizable features. Without color jitter, the network can cheat by exploiting color histograms rather than learning semantic content. The augmentation pipeline typically includes:

• Random crop and resize
• Random horizontal flip
• Color jittering (brightness, contrast, saturation, hue)
• Gaussian blur
• Optional Sobel filtering for edge-based representations

A deep convolutional neural network (typically ResNet-50 or larger variants) that extracts representation vectors from augmented images. The encoder processes each view independently and outputs a feature vector h_i = f(x_i). After pre-training, this encoder is retained for downstream tasks while the projection head is discarded. The quality of learned representations scales consistently with encoder depth and width, with ResNet-50 (4×) achieving top-1 accuracy of 76.5% on ImageNet with only 1% of labels.

A small multi-layer perceptron (MLP) with one or two hidden layers that maps encoder representations to a lower-dimensional space where the contrastive loss is applied. The architecture is typically: h → Linear → ReLU → Linear → z. Crucially, this head is discarded after pre-training—downstream evaluation uses the encoder output h directly. The projection head filters out augmentation-specific information, allowing the encoder to preserve more general semantic features in h.

The Normalized Temperature-scaled Cross Entropy Loss operates on pairs of augmented views. For a batch of N images, 2N views are generated. The loss for a positive pair (i, j) is:

• ℓ(i,j) = -log( exp(sim(z_i, z_j)/τ) / Σ_{k≠i} exp(sim(z_i, z_k)/τ) ) where sim(u, v) = u^T v / (||u|| ||v||) is cosine similarity and τ (temperature) controls concentration. The total loss is computed symmetrically across all positive pairs. Temperature tuning is critical—typical values range from 0.07 to 0.5.

Unlike MoCo which uses a momentum queue, SimCLR relies on in-batch negatives—all other 2(N-1) augmented views in the batch serve as negative examples. This requires very large batch sizes (typically 4096 to 8192) to provide sufficient negative diversity. Training with batch size 8192 on 128 TPUv3 cores was standard in the original paper. The absence of a memory bank simplifies the architecture but demands significant computational resources for effective training.

SimCLR employs the Layer-wise Adaptive Rate Scaling (LARS) optimizer to stabilize training with extremely large batch sizes. LARS computes separate learning rates for each layer based on the ratio of weight norm to gradient norm: η_l = η × ||w_l|| / (||∇L(w_l)|| + β||w_l||). This prevents gradient explosion in early layers while allowing faster updates in later layers. Combined with a cosine decay schedule and linear warmup, LARS enables stable convergence at batch sizes exceeding 4096.