Queue-Based Dictionary

The dictionary is maintained as a First-In-First-Out (FIFO) queue of encoded representations. New mini-batch encodings are enqueued while the oldest representations are dequeued, ensuring the dictionary reflects a sliding window over recent training iterations. This dynamic replacement strategy maintains a large and consistent set of negative keys without storing the entire dataset in memory.

Traditional contrastive methods like SimCLR require large batch sizes (e.g., 4096–8192) to provide sufficient negative samples, demanding massive GPU memory. The queue-based dictionary completely decouples the number of negative samples from the mini-batch size. A dictionary of 65,536 keys can be maintained while using a batch size of only 256, making unsupervised pre-training feasible on commodity hardware.

Keys in the dictionary are produced by a momentum encoder—a slowly evolving copy of the query encoder updated via exponential moving average:

• Update rule: θ_k ← mθ_k + (1-m)θ_q, where m is typically 0.999
• Benefit: Prevents rapid key fluctuation, ensuring that dictionary representations remain temporally consistent across training steps
• Result: The query can compare against a stable, slowly drifting set of negatives rather than a chaotic, rapidly changing one

Each training iteration enqueues the current mini-batch encoded by the momentum encoder and dequeues the oldest batch. This creates a progressive curriculum where:

• The oldest, most outdated representations are discarded first
• The dictionary always contains the most recent 65,536+ encoded samples
• The model is constantly challenged against increasingly difficult negatives as training progresses and representations improve

By storing only the low-dimensional encoded vectors (typically 128-d) rather than raw images or intermediate feature maps, the dictionary achieves extreme memory efficiency:

• 65,536 keys × 128 dimensions × 4 bytes ≈ 33.5 MB of GPU memory
• Enables hundreds of thousands of negatives without gradient computation through them
• The keys are treated as constant tensors during loss calculation, eliminating the need to store activations for backpropagation

The queue-based dictionary is the defining feature of the MoCo (Momentum Contrast) framework. The architecture follows a lookup-encode-enqueue cycle:

1. Lookup: Sample a large set of negative keys from the dictionary
2. Encode: Process the current query batch through the query encoder
3. Contrast: Compute InfoNCE loss against both positive keys and dictionary negatives
4. Enqueue: Add the new momentum-encoded batch to the dictionary
5. Dequeue: Remove the oldest batch to maintain fixed dictionary size