Momentum Encoder

The momentum encoder's parameters are not updated via backpropagation. Instead, they are updated as an exponential moving average (EMA) of the online encoder's weights:

• Update rule: θ_k ← m·θ_k + (1-m)·θ_q
• Momentum coefficient (m): Typically a high value like 0.999 or 0.996
• Effect: The target network evolves with high inertia, changing very slowly over training steps

This slow evolution ensures that the target representations used for contrastive learning or prediction tasks remain temporally consistent, providing a stable learning signal to the online network.

A central challenge in self-supervised learning is representational collapse, where the encoder outputs a constant or trivial vector for all inputs. The momentum encoder prevents this by:

• Asymmetric architecture: The online and target networks are not identical at any given moment, breaking symmetry
• Stop-gradient operation: Gradients flow only through the online encoder, never through the momentum encoder
• Implicit regularization: The EMA update acts as a form of temporal ensembling, smoothing the target manifold

This asymmetry is critical in frameworks like BYOL and SimSiam, where no negative pairs are used, yet collapse is avoided entirely through the stop-gradient and momentum combination.

In the MoCo (Momentum Contrast) framework, the momentum encoder serves as a key encoder for a dynamic dictionary:

• Queue-based dictionary: Stores encoded representations of recent samples as negative keys
• Consistent keys: Because the momentum encoder changes slowly, keys in the queue remain comparable across training steps
• Large-scale contrastive learning: The dictionary can be much larger than the mini-batch, enabling effective contrastive learning with thousands of negatives

The momentum encoder decouples the dictionary size from the batch size, a critical innovation that made self-supervised learning practical on commodity hardware without massive GPU memory.

For radio frequency machine learning, the momentum encoder provides unique advantages when processing sequential IQ data:

• Channel variation robustness: The slowly evolving target network is less sensitive to rapid channel fluctuations, producing stable representations across coherence time
• Consistent emitter fingerprints: When learning specific emitter identification (SEI) features, the momentum encoder smooths out transient noise while preserving hardware impairment signatures
• Contrastive predictive coding: In CPC frameworks applied to RF, the momentum encoder provides stable context vectors for predicting future latent representations of the spectrum

This temporal stability is particularly valuable in spectrum sensing and automatic modulation classification tasks where signal characteristics evolve continuously.

The momentum encoder can be understood through the lens of self-distillation:

• Teacher-student dynamic: The momentum encoder acts as the teacher, the online encoder as the student
• Polyak averaging: The teacher is a temporal ensemble of past student models, proven to produce higher-quality representations
• No separate training: Unlike traditional knowledge distillation, the teacher is updated automatically via EMA, requiring no separate training phase

This self-distillation property is the core mechanism behind BYOL and related non-contrastive methods. The student learns to predict the teacher's representation of augmented views, and the teacher's slow update ensures these targets remain meaningful throughout training.

Practical deployment of momentum encoders requires careful attention to several details:

• Momentum scheduling: Some implementations use a cosine schedule for m, starting lower (e.g., 0.996) and increasing to 0.999 during training
• Batch normalization pitfalls: Sharing BN statistics between online and momentum encoders can cause information leakage; separate BN layers or synchronized BN are common solutions
• Weight decay exclusion: Bias terms and BN parameters in the momentum encoder are typically excluded from weight decay
• Initialization: The momentum encoder is usually initialized as an exact copy of the online encoder at the start of training

These implementation details are critical for achieving stable convergence in self-supervised RF pre-training pipelines.

The mathematical backbone of the momentum encoder. Instead of backpropagation, the momentum encoder's parameters θ_k are updated as: θ_k ← m·θ_k + (1−m)·θ_q, where m is the momentum coefficient (typically 0.999). This slow, smooth evolution ensures the target representations change gradually, providing stable learning targets for the query encoder. A high m value prioritizes consistency over recency, which is critical for preventing the target network from collapsing to a trivial solution.

A critical architectural component that blocks gradient flow to the momentum encoder. In frameworks like BYOL and SimSiam, the stop-gradient prevents the model from finding a trivial collapsed solution where both encoders output identical constant vectors. By only updating the target network through EMA, the stop-gradient creates an asymmetric learning dynamic that forces the online network to learn meaningful representations without the need for negative pairs or large batches.

The primary failure mode that momentum encoders are designed to prevent. Collapse occurs when the encoder produces a constant or non-informative output for all inputs, achieving zero loss trivially. Momentum encoders combat this by: (1) providing a slowly evolving target that prevents the online network from exploiting shortcut solutions, and (2) maintaining a diverse set of representations in the dynamic dictionary. Variance and covariance regularization are complementary techniques that explicitly penalize collapsed embeddings.

The broader learning framework in which momentum encoders operate. The teacher (momentum encoder) provides soft targets for the student (query encoder) without backpropagation through the teacher. This is distinct from standard knowledge distillation because: (1) the teacher and student share identical architectures, (2) the teacher is not pre-trained but evolves alongside the student, and (3) no labeled data is required. The EMA update ensures the teacher is a temporally ensembled version of the student.