MoCo

A slowly evolving copy of the main query encoder, updated via exponential moving average (EMA) rather than backpropagation. In MoCo, the momentum encoder generates consistent, stable keys for the dynamic dictionary. The update rule is: θ_k ← mθ_k + (1 − m)θ_q, where m is typically a large momentum coefficient like 0.999. This slow evolution ensures that keys from different mini-batches are comparable, preventing the dictionary from being corrupted by rapidly changing representations. Without the momentum mechanism, the queue-based dictionary would contain inconsistent keys, degrading the contrastive learning signal.

Noise Contrastive Estimation loss adapted for mutual information maximization. The objective is to pull an anchor query and its positive key together while pushing apart K negative keys stored in the queue. Formally: L_q = −log[exp(q·k_+/τ) / Σ_{i=0}^K exp(q·k_i/τ)], where τ is a temperature hyperparameter controlling concentration. A lower τ (e.g., 0.07) produces sharper distributions, penalizing hard negatives more heavily. This loss directly optimizes a lower bound on mutual information between different views of the same underlying signal, making it ideal for learning representations from unlabeled IQ streams.

MoCo's key innovation: decoupling dictionary size from mini-batch size using a first-in-first-out (FIFO) queue. The queue stores encoded representations from the most recent K mini-batches, enabling large-scale negative sampling without massive GPU memory requirements. As each new mini-batch is processed, its encoded keys are enqueued while the oldest mini-batch is dequeued. This design achieves dictionary sizes of 65,536 or more, compared to the few thousand achievable with end-to-end contrastive methods like SimCLR. For RF applications, this large negative set is critical for learning discriminative features that distinguish subtle modulation differences.

A catastrophic failure mode where the encoder produces a constant or degenerate output for all inputs, achieving zero contrastive loss trivially. In MoCo, collapse is prevented by the stop-gradient operation on the momentum encoder—gradients flow only through the query encoder, while the key encoder is updated solely via EMA. This architectural asymmetry breaks the symmetry that would otherwise allow both encoders to converge to identical trivial solutions. Additional safeguards include variance regularization and covariance regularization, which explicitly penalize low embedding diversity and feature redundancy respectively.

A small multi-layer perceptron (MLP) attached to the backbone encoder during pre-training, mapping representations to a lower-dimensional space where contrastive loss is applied. The projection head typically has 2-3 layers with a ReLU hidden activation and no output nonlinearity. Crucially, this head is discarded after pre-training—downstream tasks use the frozen backbone features directly. The projection head learns to strip away augmentation-specific information, preserving only the semantic content relevant to instance discrimination. For IQ data, this means the backbone retains modulation and emitter characteristics while discarding channel-specific artifacts.

A precursor to MoCo that learns representations by predicting future latent representations from past context using a probabilistic contrastive loss. While MoCo contrasts different augmented views of the same instance, CPC operates on temporal sequences, making it naturally suited for raw IQ time-series data. The CPC architecture uses an autoregressive model (e.g., GRU) to summarize past context into a context vector, then maximizes mutual information between this context and future encoded signal patches. Both methods share the InfoNCE loss foundation but differ in their pretext task design—instance discrimination versus future prediction.