Contrastive Predictive Coding (CPC)

CPC employs an autoregressive model (typically a GRU or causal Transformer) to summarize the history of latent representations into a single context vector (c_t). This context vector is not a simple average; it encodes the sequential dependencies necessary to predict future states. The model is trained to make these predictions in a compressed latent space rather than the raw observation space, forcing it to discard low-level noise and retain high-level semantic information. This is critical for medical imaging sequences (e.g., ultrasound video or dynamic contrast-enhanced MRI) where temporal evolution carries diagnostic significance.

Instead of directly modeling the high-dimensional distribution of future observations p(x_{t+k} | c_t), CPC estimates a density ratio that preserves the mutual information between the context and future latents. The model uses the InfoNCE (Noise-Contrastive Estimation) loss, which frames the prediction task as a categorical classification problem. The objective is to correctly identify the true future latent (positive sample) from a set of randomly sampled negative latents. This probabilistic contrastive approach avoids the computational intractability of generative modeling while learning representations that maximize mutual information.

The theoretical foundation of CPC is the maximization of mutual information (MI) between the context vector c_t and future latent representations z_{t+k}. The InfoNCE loss provides a lower bound on MI: I(x; c) ≥ log(N) - L_N, where N is the number of negative samples. By optimizing this bound, CPC learns to extract the underlying shared information between temporally separated signals. In medical imaging, this translates to learning features that capture anatomical and pathological structures invariant to short-term temporal variations, such as cardiac or respiratory motion.

The quality of CPC representations depends heavily on the negative sampling strategy. For a given positive pair (c_t, z_{t+k}), negative samples are drawn from other time steps within the same sequence or from entirely different sequences. Hard negative mining—selecting negatives that are semantically similar but temporally incorrect—forces the model to learn finer-grained distinctions. In medical contexts, this could mean distinguishing between visually similar tissue textures that appear at different phases of a contrast-enhanced scan, improving the model's sensitivity to subtle pathological changes.

CPC operates in a compressed latent space rather than raw pixel space. A non-linear encoder g_enc maps observations x_t to latent representations z_t. The autoregressive model g_ar then summarizes past latents into a context c_t. A log-bilinear model f_k(c_t, z_{t+k}) = exp(z_{t+k}^T W_k c_t) scores the compatibility between context and future latents. This separation of concerns—encoding, context aggregation, and scoring—allows each component to specialize. The encoder learns to extract meaningful features, the autoregressive model captures temporal dynamics, and the scoring function evaluates predictive accuracy.

CPC predicts multiple future time steps simultaneously using separate prediction heads W_k for each offset k. This multi-step objective forces the context vector c_t to encode information relevant across different temporal horizons, from immediate next-step features to longer-range semantic structure. The model learns a hierarchy of temporal representations: short-term predictions capture local dynamics, while long-term predictions require understanding global context. For medical time-series like EEG or functional MRI, this enables the extraction of features spanning multiple temporal scales relevant to neurological diagnosis.

The parent paradigm of CPC. This framework learns representations by pulling positive pairs (e.g., adjacent signal windows) together and pushing negative pairs (e.g., random, non-sequential samples) apart in embedding space.

• CPC's Role: Applies this principle to sequential data using autoregressive context.
• Medical Use: Distinguishes between normal anatomical progression in scans and pathological anomalies.

The Noise-Contrastive Estimation objective that mathematically defines CPC's training. It maximizes the mutual information between a context vector and future observations by framing the task as a classification problem over a set of negative samples.

• Mechanism: Uses a log-bilinear model to score the compatibility of predicted and actual future latents.
• Impact: Provides a probabilistic metric that is more stable than direct density estimation for high-dimensional medical volumes.

The sequential modeling backbone used in CPC to summarize past observations into a single context vector. Typically implemented via a recurrent neural network (RNN) or a causal transformer.

• Function: Aggregates latent representations from previous time steps or image slices.
• Medical Context: Essential for modeling the sequential nature of 3D volumetric scans (CT/MRI) where each slice is dependent on the anatomical context of the preceding slice.

The strategy of selecting dissimilar data points to serve as repulsive anchors in the InfoNCE loss. The quality of CPC representations heavily depends on the sophistication of this sampling.

• Hard Negative Mining: Selecting patches from the same organ but a different patient to force the model to learn fine-grained pathological details.
• Challenge: Poor sampling can lead to representation collapse or trivial solutions that fail to capture clinical variance.

An alternative self-supervised family (e.g., MAE) that reconstructs hidden patches rather than predicting future latents. While CPC uses a predictive objective, MIM uses a generative reconstruction objective.

• Comparison: CPC excels at learning long-range sequential dependencies; MIM excels at learning fine-grained local texture.
• Synergy: Hybrid approaches combine CPC's global context with MIM's local detail for robust whole slide image analysis.

The standardized benchmark used to validate CPC pre-training. A frozen backbone extracts features, and a single linear classifier is trained on top to measure representation quality.

• Metric: Top-1 accuracy on downstream tasks like pathology classification.
• Relevance: Proves that CPC-learned features are linearly separable without fine-tuning, indicating a well-structured latent space for biomarker identification.