Contrastive Predictive Coding (CPC)

CPC uses an autoregressive model, typically a Gated Recurrent Unit (GRU) or causal Transformer, to summarize the history of past latent representations into a single compact context vector. This context vector encodes all relevant information from the past signal sequence, enabling the model to make informed predictions about future signal states without requiring a hand-crafted state-space model.

Instead of directly modeling the high-dimensional distribution of future signals, CPC learns to estimate a density ratio that preserves the mutual information between the context and future observations. The model is trained with InfoNCE (Noise Contrastive Estimation) loss, which scores the true future latent as a positive sample against a set of negative samples drawn from the proposal distribution. This forces the encoder to capture the underlying slow features that are shared across time.

The CPC objective naturally encourages the encoder to learn slowly varying features that are predictable over long temporal horizons. In the RF domain, this translates to learning representations that are invariant to fast-varying channel noise while capturing stable signal characteristics such as:

• Modulation scheme (QPSK, 16QAM)
• Symbol rate and pulse shaping
• Hardware-specific impairments for fingerprinting This makes CPC ideal for pre-training on unlabeled spectrum captures.

The quality of CPC representations is highly dependent on the negative sampling strategy. Hard negative mining—selecting negatives that are temporally close or from the same signal class—improves the discriminative power of the learned features. In RF applications, negatives can be sampled from:

• Other time steps in the same batch
• Different frequency bands
• Synthetic IQ samples from a GAN This prevents the model from exploiting trivial shortcuts.

After self-supervised pre-training on massive unlabeled RF datasets, the trained encoder backbone is frozen or fine-tuned for downstream tasks with limited labeled data. CPC-pretrained models have demonstrated state-of-the-art performance in few-shot modulation recognition, specific emitter identification, and spectrum anomaly detection, often surpassing fully supervised baselines when labeled data is scarce.

The theoretical foundation of CPC is the maximization of mutual information (MI) between the context vector and future observations. By optimizing the InfoNCE bound on MI, the model learns representations that retain maximal information about the signal's future state while discarding unpredictable noise. This principle directly combats the low signal-to-noise ratio (SNR) challenges inherent in real-world RF environments.

The probabilistic contrastive loss that powers CPC. It maximizes the mutual information between context vectors and future latent representations by distinguishing a positive sample from a set of negative samples.

• Formulated as a multi-class classification problem over N+1 candidates
• The density ratio is estimated using a log-bilinear model
• Lower loss indicates the model is learning to predict future structure rather than memorizing noise

A slowly evolving copy of the main encoder, updated via exponential moving average (EMA), used to produce consistent target representations for contrastive learning.

• Prevents representation collapse by providing stable targets
• Critical in frameworks like MoCo and BYOL
• In RF applications, ensures consistent embeddings despite channel variations

A critical failure mode where the encoder produces a constant or non-informative output for all inputs, rendering the learned representations useless.

• Often prevented by variance regularization and covariance regularization
• The stop-gradient operation in self-distillation frameworks explicitly blocks this shortcut
• Detected by monitoring the standard deviation of embeddings across a batch

A small MLP module attached to the backbone encoder during self-supervised pre-training. It maps representations to a space where the contrastive loss is applied.

• Discarded after pre-training before downstream fine-tuning
• Prevents the loss from forcing the backbone to discard useful information
• Typically a 2-3 layer network with a bottleneck architecture

The process of training a neural network on a large unlabeled dataset using a pretext task to learn general-purpose representations.

• CPC uses future prediction as the pretext task
• After pre-training, the model is fine-tuned on a smaller labeled downstream task
• Enables few-shot modulation recognition where only 5-10 labeled examples per class are available

The ability of a model trained on source RF environments to perform accurately on unseen target domains with different channel conditions or hardware impairments.

• CPC's focus on slow features that persist over time promotes domain invariance
• Contrastive objectives naturally learn representations robust to nuisance variables
• Critical for deploying models across different receivers and spectrum bands without retraining