Exponential Moving Average Update

The EMA update is defined as θ_t ← mθ_t + (1-m)θ_s, where θ_t represents the target network weights, θ_s represents the source (online) network weights, and m is the momentum coefficient. This creates a weighted blend where the target network retains a large fraction of its own history while slowly incorporating new knowledge. The momentum coefficient m is typically set very high, often 0.99 or 0.999, ensuring the target network evolves smoothly across thousands of training steps.

A critical architectural component in frameworks like BYOL and DINO, the EMA update is paired with a stop-gradient operation on the target branch. Gradients flow only through the online network; the target network is updated exclusively via EMA. This breaks the symmetry between the two branches, preventing the model from finding a trivial collapsed solution where both encoders output identical constant vectors. Without this asymmetry, non-contrastive methods would fail to learn meaningful representations.

The EMA update effectively creates a temporal ensemble of the online network's weights over recent training history. By maintaining a high momentum coefficient, the target network represents a polyak-averaged version of the model, smoothing out the noise from individual gradient steps. This ensemble effect produces more stable and consistent target representations, which serve as higher-quality regression targets for the online network. The result is improved convergence and final representation quality compared to using instantaneous online weights.

In self-distillation methods like DINO and BYOL, the EMA-updated teacher network generates pseudo-labels or target projections that the student network must match. The teacher's slow evolution prevents the target from changing too rapidly, which would destabilize the student's learning. Key benefits include:

• Consistent learning signal across consecutive batches
• Implicit knowledge distillation from the temporal ensemble
• Avoidance of confirmation bias where the student chases a rapidly shifting target

While a constant momentum coefficient is common, advanced implementations use a cosine schedule that increases m from an initial value (e.g., 0.996) to a final value (e.g., 1.0) over the course of training. This strategy allows the target network to adapt more quickly during early training when representations are changing rapidly, then become increasingly stable as the model converges. The schedule is typically defined as: m ← 1 - (1 - m_base) × (cos(πk/K) + 1) / 2, where k is the current step and K is the total steps.

The EMA update for target networks should not be confused with Batch Normalization (BN) momentum, which controls the running mean and variance statistics used during inference. While both use exponential moving averages, they operate on fundamentally different objects:

• Target network EMA: Updates model weights across training steps
• BN momentum: Updates activation statistics within a single forward pass
• Target network EMA uses very high momentum (0.99+); BN momentum typically uses lower values (0.1–0.9)

The momentum encoder is the direct architectural instantiation of the EMA update rule. It is a slowly evolving copy of the primary online network, updated via θ_target ← mθ_target + (1-m)θ_online. This temporal ensembling produces stable, consistent target representations that prevent the model from collapsing to trivial solutions in frameworks like BYOL and MoCo. The momentum coefficient m is typically set very high (e.g., 0.999) to ensure smooth evolution.

BYOL is the canonical non-contrastive framework that relies entirely on the EMA update to function. It trains an online network to predict the target representations of a target network, which is updated via EMA. Critically, BYOL does not use negative pairs; the slowly moving average of weights acts as an implicit regularization mechanism. The stop-gradient operation on the target branch, combined with EMA, breaks symmetry and prevents representation collapse.

MoCo uses an EMA update to maintain a consistent key encoder that produces representations for a dynamic dictionary queue. The slow evolution of the key encoder via θ_k ← mθ_k + (1-m)θ_q ensures that keys in the queue remain consistent over time, even as the query encoder updates rapidly. This decouples the dictionary size from the mini-batch size, enabling large-scale contrastive pre-training with a massive set of negative samples.

Self-distillation is the broader learning paradigm that EMA-based frameworks instantiate. The online network (student) is trained to predict the outputs of the momentum-updated network (teacher). This transfers dark knowledge from a temporally averaged version of the model to itself, smoothing the learning signal. The teacher's weights are not learned via backpropagation but are a polyak-averaged version of the student's weights, creating a bootstrapped curriculum.

The stop-gradient operation is a critical companion to the EMA update in non-contrastive methods. It prevents gradients from propagating through the target network branch, ensuring that the EMA-updated encoder is treated as a fixed regression target. Without this operation, the siamese network would suffer from representation collapse, where both branches trivially agree by outputting constant vectors. The combination of stop-gradient and EMA is the core symmetry-breaking mechanism.

Representation collapse is the primary failure mode that EMA update rules are designed to prevent. It occurs when an encoder learns to output a constant or non-informative vector for all inputs, achieving a perfect loss without capturing meaningful data variance. In joint embedding architectures, the EMA update on the target network introduces an asymmetry that makes collapse an unstable solution. The online network must continuously adapt to match a slowly shifting target, forcing it to learn useful features.