Contrastive Predictive Coding (CPC)

CPC employs a recurrent neural network (often a GRU) as an autoregressive model to summarize the history of past latent representations into a single compact context vector.

• The context vector aggregates information from all previous timesteps.
• It discards low-level noise and retains high-level, slowly varying features useful for prediction.
• This forces the model to capture the underlying structure of the data rather than trivial local statistics.

Instead of predicting exact future values, CPC maximizes the mutual information between the context vector and future observations using a contrastive loss called InfoNCE.

• A density ratio is estimated via a log-bilinear model: f_k(x_{t+k}, c_t) = exp(z_{t+k}^T W_k c_t).
• The model is trained to score the true future sample higher than a set of negative samples drawn from a proposal distribution.
• This avoids costly generative modeling of high-dimensional data and focuses on capturing shared information.

CPC learns representations that vary slowly over time, aligning with the principle that high-level semantic content changes more gradually than raw signal values.

• A non-linear encoder maps raw observations to a lower-dimensional latent space.
• The autoregressive model is trained to predict future latents up to K steps ahead.
• This encourages the encoder to filter out high-frequency noise and retain features that persist across multiple timesteps, making it ideal for speech, video, and RF signal analysis.

CPC can be extended to learn representations at multiple temporal scales by applying the contrastive loss to predictions at various offsets simultaneously.

• Separate context networks can be trained for different prediction horizons.
• This captures both short-term dynamics (e.g., phoneme transitions) and long-term structure (e.g., speaker identity).
• In automatic modulation classification, this allows the model to learn both rapid symbol-rate features and slower frame-level protocol structures from raw IQ samples.

The representations learned by CPC serve as powerful, general-purpose features that transfer effectively to supervised tasks with limited labeled data.

• After pre-training on unlabeled data, the encoder can be frozen or fine-tuned for classification.
• CPC features have achieved state-of-the-art results in phoneme recognition, speaker identification, and image classification without using any labels during pre-training.
• For few-shot modulation learning, a CPC encoder pre-trained on raw, unlabeled spectrum captures provides a robust initialization, drastically reducing the number of labeled examples needed to identify novel signal types.

The quality of CPC representations depends heavily on the choice of negative samples used in the InfoNCE loss.

• Negatives can be drawn uniformly from the same sequence or from other sequences in the batch.
• Hard negative mining—selecting negatives that are most similar to the positive—improves the discriminative power of the learned features.
• In RF applications, negatives can be sampled from signals with different modulation schemes, forcing the model to learn features that are invariant to noise but discriminative across modulation types.

The probabilistic contrastive loss at the heart of CPC. It uses noise-contrastive estimation to maximize the mutual information between a context vector and future latent representations. The model must correctly identify the true future sample from a set of negative distractors.

• Mechanism: A categorical cross-entropy loss over a batch of samples
• Objective: Pulls positive pairs (context, future) together; pushes negatives apart
• Key property: The density ratio estimated by the loss preserves the mutual information between the signal's past and future

CPC employs a unidirectional recurrent or masked convolutional network to summarize the history of latent representations into a single compact context vector. This context is then used to predict multiple future time steps.

• Common architectures: Gated Recurrent Units (GRUs) or causal Transformers
• RF application: Summarizes the sequential structure of IQ samples to capture the temporal dynamics of a modulation scheme
• Contrast with: Bidirectional models like BERT, which are not suitable for predictive tasks

The theoretical underpinning of CPC. Rather than predicting raw signal values, CPC maximizes the mutual information between the context representation and future observations. This encourages the model to extract slow features—high-level characteristics that persist over time.

• Benefit: Filters out low-level noise and focuses on semantically meaningful signal structure
• Outcome: Learned representations are excellent for downstream tasks like modulation classification with very few labels

The choice of negative samples critically impacts representation quality. Effective strategies for RF signals include:

• Temporal negatives: Samples from other time steps in the same signal recording
• Cross-signal negatives: Samples from different modulation types or noise-only segments
• Hard negative mining: Selecting negatives that are most easily confused with the positive, forcing the model to learn more discriminative features for similar constellations like QAM16 vs. QAM64

CPC is a pretext task—a self-supervised objective whose byproduct is a useful representation. For modulation recognition, effective pretext tasks include:

• Future latent prediction: Standard CPC applied to raw IQ sequences
• Temporal shuffling detection: Identifying whether signal segments are in the correct order
• Contrastive multiview coding: Learning representations that are invariant to known channel impairments by contrasting differently augmented views of the same signal

The standard benchmark for self-supervised representation quality. After CPC pre-training on unlabeled signals, a frozen feature extractor is used to generate embeddings. A simple linear classifier is then trained on top using a small labeled support set.

• Metric: Classification accuracy with limited labels (e.g., 5-shot)
• Relevance: Directly measures how well CPC representations enable few-shot modulation learning without fine-tuning the entire network