MoCo

MoCo treats contrastive learning as dictionary look-up. It maintains a dynamic dictionary as a queue of encoded keys from preceding mini-batches.

• Queue Size: Decoupled from the mini-batch size, allowing the dictionary to be arbitrarily large (e.g., 65,536 keys).
• FIFO Mechanism: The oldest mini-batch is dequeued, and the newest is enqueued, ensuring consistency over recent history.
• Benefit: Provides a rich set of negative samples without requiring enormous GPU memory for large batch sizes.

The key encoder is not trained via backpropagation. Instead, its parameters are an exponential moving average (EMA) of the query encoder's parameters.

• Update Rule: θ_k ← mθ_k + (1 − m)θ_q, where m is a large momentum coefficient (e.g., 0.999).
• Consistency: The slow evolution ensures that keys in the queue are encoded by highly similar encoders, preventing stale representations.
• No Gradient: The momentum encoder does not receive gradients, breaking the end-to-end backpropagation path for keys.

MoCo uses InfoNCE loss to train the query encoder. For each query q, there is a single positive key k+ and a large set of negative keys {k−} from the queue.

• Objective: Maximize the similarity between q and k+ while minimizing similarity with all keys in the queue.
• Temperature: A temperature parameter τ controls the concentration of the distribution, sharpening or softening the penalty on hard negatives.
• Score: The logit for the positive pair is compared against the sum of logits for all negative pairs via softmax cross-entropy.

Unlike SimCLR, which relies on large batch sizes to provide sufficient in-batch negatives, MoCo decouples the dictionary size from the batch size.

• Small Batch Training: MoCo can train effectively with a batch size of 256 or even smaller.
• Scalability: The queue can hold millions of keys, providing a vast negative pool regardless of GPU constraints.
• Practicality: This makes MoCo accessible for researchers without access to massive TPU pods or large GPU clusters.

MoCo addresses a subtle cheating issue where the model exploits Batch Normalization (BN) statistics to find a trivial solution rather than learning semantic features.

• Problem: BN across GPUs leaks information between positive and negative samples, allowing the model to distinguish them without learning content.
• Solution: Shuffle the order of keys across GPUs before feeding them to the momentum encoder, then unshuffle the encoded keys.
• Effect: This ensures BN statistics are computed on independent sub-batches, preventing intra-batch communication.

MoCo v2 integrates architectural improvements from SimCLR while retaining the momentum queue framework.

• MLP Projection Head: Replaces the single linear layer with a 2-layer MLP with ReLU activation, improving representation quality.
• Stronger Augmentation: Adds Gaussian blur to the augmentation pipeline alongside random cropping and color jittering.
• Cosine Learning Rate Schedule: Adopts a cosine decay schedule for more stable convergence.
• Result: Outperforms SimCLR with significantly smaller batch sizes.

A slowly evolving copy of the main encoder network, updated via exponential moving average (EMA) rather than backpropagation. In MoCo, the momentum encoder generates consistent representations for the negative samples stored in the dynamic dictionary, preventing rapid feature drift that would invalidate the queue. The update rule follows: θ_k ← mθ_k + (1 − m)θ_q, where m is typically a large momentum coefficient like 0.999. This ensures the keys remain temporally coherent across training steps.

MoCo's core innovation: a first-in-first-out (FIFO) queue that decouples dictionary size from mini-batch size. The queue stores encoded representations from preceding mini-batches, allowing the model to contrast against thousands of negative samples without enormous GPU memory requirements. As new mini-batches are processed, the oldest keys are dequeued, maintaining a consistent and evolving set of negatives. This design enables effective contrastive learning on commodity hardware with batch sizes as small as 256.

Information Noise-Contrastive Estimation, the objective function at the heart of MoCo. It formulates representation learning as a categorical cross-entropy problem: given a query q and a set of keys {k₀, k₁, ..., k_K} containing one positive match k₊, the model must identify the correct pair. The loss maximizes the mutual information between query and positive key representations while pushing negatives away. The temperature parameter τ controls the concentration of the softmax distribution, with lower values imposing harder penalties on confusing negatives.

A Simple Framework for Contrastive Learning of Visual Representations, the primary architectural rival to MoCo. SimCLR dispenses with the momentum encoder and memory queue entirely, instead relying on extremely large batch sizes (up to 8,192) to provide sufficient in-batch negatives. It uses a projection head MLP to map representations before applying the NT-Xent loss. While simpler to implement, SimCLR demands significant GPU memory and scales poorly on limited hardware, making MoCo the preferred choice for resource-constrained training environments.

Bootstrap Your Own Latent, a framework that challenges the necessity of negative pairs entirely. BYOL trains an online network to predict the output of a target momentum network using an additional predictor MLP. The stop-gradient operation on the target branch prevents representation collapse without any contrastive negatives. This demonstrates that the momentum update mechanism—shared with MoCo—is a powerful regularizer in its own right, capable of stabilizing self-supervised learning even in purely positive-pair regimes.

The strategic selection of deceptively similar but semantically distinct samples to serve as negatives. In MoCo's queue-based dictionary, hard negatives naturally emerge as the encoder improves and the queue fills with increasingly sophisticated representations. Effective hard negatives force the model to learn fine-grained discriminative features rather than exploiting trivial differences. Without explicit mining, contrastive methods risk learning shortcuts that fail to generalize to semantically challenging retrieval scenarios.