Momentum Encoder

The momentum encoder's parameters are updated via an exponential moving average (EMA) of the primary encoder's weights, not by backpropagation. The update rule is: θ_k ← mθ_k + (1 − m)θ_q, where m is the momentum coefficient (typically 0.999). This ensures the encoder evolves slowly and smoothly, preventing abrupt changes that would make older representations in the queue inconsistent with newer ones.

A large, dynamic queue stores encoded representations of negative samples from recent mini-batches. The momentum encoder ensures that keys encoded at different times remain comparable in the same embedding space. Without momentum, the rapidly changing primary encoder would make older queue entries stale, degrading the quality of contrastive learning by introducing false negatives.

The architecture separates the query encoder (trained via gradient descent) from the key encoder (updated via momentum). This decoupling is critical: the query encoder learns to pull relevant items closer, while the key encoder provides a stable target representation. This design originates from MoCo (Momentum Contrast) and is foundational to self-supervised visual and textual representation learning.

The momentum encoder treats contrastive learning as a dynamic dictionary look-up task. The encoded query is matched against a dictionary of encoded keys. The goal is to maximize similarity with the positive key while minimizing similarity with all negative keys in the queue. This formulation allows the dictionary to be much larger than the mini-batch size, improving the richness of the contrastive signal.

In DPR training, a momentum encoder is often used to maintain a large cache of passage embeddings as negatives. The passage encoder is updated via momentum while the query encoder is trained normally. This enables efficient in-batch and cross-batch negative sampling without recomputing all passage embeddings at every step, dramatically scaling up the number of negatives used for contrastive loss.

A key benefit of the momentum encoder is preventing representation collapse, where all embeddings converge to a trivial constant vector. By maintaining a slowly evolving target network, the model is forced to learn meaningful, spread-out representations. The large queue of diverse negatives further regularizes the embedding space, ensuring that semantically distinct items remain well-separated.