Exponential Moving Average (EMA)

The momentum encoder's parameters are not trained by backpropagation. Instead, they are updated as an exponential moving average of the online encoder's weights: θ_k ← m*θ_k + (1-m)*θ_q. The momentum coefficient (m) is typically set very high, such as 0.999 or 0.996, ensuring the target network aggregates information over many training steps. This acts as a form of temporal ensembling, where the teacher model represents a stable, polyak-averaged version of the rapidly changing student.

The high momentum value ensures that the target representation changes very slowly between consecutive batches. This temporal smoothness is critical for learning from unlabeled RF data, where channel conditions and noise floors may vary. The EMA provides a consistent target view, allowing the online network to learn invariances to short-term perturbations without being distracted by a rapidly jittering target. This is particularly vital for few-shot modulation recognition tasks where stability is paramount.

The momentum coefficient is often scheduled during training. A common strategy is to use a cosine schedule that increases m from a base value (e.g., 0.996) to a higher value (e.g., 1.0) during training. This allows the teacher to adapt faster in the early stages when the student is learning rapidly, and become more stable later. For RF applications with highly non-stationary data, a constant high momentum is preferred to maintain a stable prototypical representation of emitters.

The momentum encoder parameters θₖ are updated as an exponential moving average of the online encoder parameters θₚ:

θₖ ← m · θₖ + (1 - m) · θₚ

• m (momentum coefficient) is typically close to 1, e.g., 0.996 or 0.999
• The target network evolves smoothly and slowly, acting as a temporal ensemble of past online networks
• Gradients flow only through the online network; the momentum encoder is updated via weight copying, not backpropagation
• This decouples the target from the current noisy update, preventing the model from finding a trivial collapsed solution

The stop-gradient operation is the architectural complement to EMA updates. It blocks gradient flow into the momentum encoder, forcing the online network to predict the teacher's outputs without the teacher adapting to the student.

• In BYOL, the online network predicts the target network's projection of an augmented view
• In MoCo, the momentum encoder produces consistent keys for the dynamic dictionary
• The combination of EMA + stop-gradient creates an asymmetric learning dynamic where the teacher is always a slightly delayed, more stable version of the student
• Without stop-gradient, both networks would collapse to a constant representation

For self-supervised RF representation learning, EMA-based momentum encoders are critical due to the high variability of wireless signals:

• Channel fading and noise create natural augmentations; the EMA teacher provides stable targets despite these perturbations
• When pre-training on unlabeled IQ samples, the momentum encoder builds a consistent feature space for modulation types, emitter identities, and signal structures
• Temporal consistency is enforced: the same emitter captured at different times should map to similar representations
• EMA coefficients are often tuned higher (0.999+) for RF data to smooth over rapid channel variations while preserving discriminative signal features

Representation collapse occurs when the encoder outputs a constant vector for all inputs, achieving zero loss but learning nothing. EMA-based architectures prevent this through several mechanisms:

• Asymmetric architecture: The online network has a predictor head that the target network lacks, breaking symmetry
• Slow teacher evolution: The EMA teacher cannot instantly adapt to match the student's output, forcing the student to learn meaningful features
• Implicit contrastive effect: The student must predict the teacher's representation, which acts as an implicit negative sample against collapsed outputs
• Alternative collapse prevention methods include variance regularization (VICReg) and cross-correlation decorrelation (Barlow Twins), but EMA remains the dominant approach in BYOL and MoCo

Different self-supervised frameworks handle target network updates differently:

• EMA (BYOL, MoCo v2/v3): Smooth parameter blending with high momentum; no negative pairs required
• Shared weights (SimCLR): Both branches use the same encoder updated by backpropagation; requires large negative sample batches
• Stop-gradient only (SimSiam): Demonstrates that EMA is not strictly necessary—stop-gradient alone can prevent collapse with a predictor MLP
• No target network (Barlow Twins, VICReg): Collapse prevented through explicit regularization terms on the embedding covariance matrix
• For RF few-shot learning, EMA-based methods often outperform alternatives because they build more stable prototypes from limited labeled examples

When implementing EMA for RF self-supervised learning:

• Momentum schedule: Start with a lower momentum (0.99) and cosine-anneal to a higher value (0.999+) during training
• Batch normalization: The momentum encoder should use the online network's running statistics, not its own, to avoid information leakage
• Weight decay: Apply only to the online network; the momentum encoder inherits regularized weights through the EMA update
• Sync frequency: Update the momentum encoder every training step, not every epoch, for smooth evolution
• FP16 considerations: Use parameter-level EMA rather than buffer-level to maintain precision in mixed-precision training

A critical failure mode in self-supervised learning that the EMA mechanism helps prevent. Collapse occurs when the encoder produces a constant, non-informative output for all inputs, cheating the loss function without learning useful features. In contrastive methods, a slowly updating momentum encoder provides a diverse set of negative keys in a dynamic dictionary, preventing the model from finding a trivial solution.

• Prevention: EMA ensures a stable but evolving target.
• Related: Variance Regularization and Covariance Regularization explicitly penalize collapse.
• Symptom: Training loss drops rapidly, but downstream accuracy is near random.

A critical architectural component that works in tandem with the EMA update. In frameworks like BYOL and SimSiam, the stop-gradient operation blocks gradient flow to the target (teacher) network. The teacher's parameters are updated solely via the EMA of the online network's weights, not by direct optimization. This asymmetric update prevents the twin networks from finding a collapsed solution where both outputs become identical.

• Mechanism: sg(θ_teacher) prevents gradient updates.
• Purpose: Enforces learning by prediction, not mirroring.
• Synergy: EMA provides the slow parameter evolution; stop-grad prevents collapse.

A knowledge distillation paradigm where the teacher and student share the same architecture, and the teacher's weights are an Exponential Moving Average of the student's. The student is trained to predict the teacher's output on augmented views of the data. This bootstrapping process, central to BYOL and DINO, allows the model to learn high-quality representations without any negative pairs.

• Process: Student learns to match the EMA teacher's output.
• Key Insight: The EMA acts as a temporal ensemble, smoothing out student noise.
• Result: Emerges with semantically meaningful features without explicit contrastive loss.