Contrastive Predictive Coding

CPC employs an autoregressive model—typically a GRU or causal CNN—to aggregate past latent representations into a single dense context vector. This context summarizes all relevant historical information up to time step t, enabling the model to discard low-level noise and retain high-level semantics. The context vector is then used to predict future latent representations multiple steps ahead, forcing the encoder to capture slow-moving features that span long temporal horizons.

The core objective is the InfoNCE loss, a categorical cross-entropy that maximizes the mutual information between the context vector and future observations. For each positive pair (context, future latent), the model must identify the true future among a set of negative samples drawn from other time steps or sequences. This density ratio estimation avoids directly modeling the complex high-dimensional data distribution, instead learning a similarity function that scores true pairs higher than impostors.

A non-linear encoder g_enc maps raw observations x_t into a lower-dimensional latent representation z_t. This compressed space discards irrelevant local details, retaining only features useful for predicting future latents. By operating in latent space rather than raw observation space, CPC avoids wasting capacity on pixel-level or waveform-level reconstruction, focusing entirely on the shared information between past and future that defines the underlying structure of the data.

CPC constructs negative samples by drawing latent representations from other time steps within the same sequence or from entirely different sequences. This in-batch negative approach creates a N-way classification problem where the model must discriminate the true future from N-1 distractors. The difficulty of the task scales with the number of negatives, and careful sampling—such as avoiding temporally adjacent negatives—prevents the model from exploiting trivial shortcuts.

Unlike single-step prediction, CPC predicts latent representations up to k steps ahead using separate linear projection heads for each horizon. This multi-step objective forces the context vector to encode information that persists across extended temporal windows, filtering out transient phenomena. The shared encoder and autoregressive model must therefore extract features with slow temporal dynamics, which often correspond to phonemes in speech, objects in video, or motifs in time-series.

The theoretical foundation of CPC is the maximization of mutual information I(x; c) between future observations and the context vector. The InfoNCE loss provides a lower bound on this mutual information: I(x; c) ≥ log(N) - L_InfoNCE, where N is the number of negative samples. This bound becomes tighter as N increases, explaining why CPC benefits from large batch sizes or memory banks that provide more negative examples for more accurate density ratio estimation.

The Information Noise-Contrastive Estimation loss is the mathematical engine of CPC. It frames representation learning as a categorical cross-entropy problem: given a context vector and a set of future latent representations, the model must identify the true positive sample among a batch of negative distractors. This directly maximizes the mutual information between context and future observations, with the density ratio estimated by a log-bilinear model. The loss lower-bounds mutual information, making it a principled objective for unsupervised feature extraction from high-dimensional sequential data like audio waveforms.

CPC employs a unidirectional recurrent or masked convolutional network to aggregate past latent representations into a compact context vector. This autoregressive model—typically a GRU or causal CNN—ensures that the context at timestep t summarizes all information up to that point without peeking into the future. The context vector serves as the conditioning signal for predicting multiple steps ahead. The choice of architecture involves a trade-off: recurrent networks provide infinite receptive fields but suffer from vanishing gradients, while dilated causal convolutions offer stable training and parallel computation across time.

The theoretical goal of CPC is to extract shared information between context vectors and future observations. Mutual information quantifies the reduction in uncertainty about future latent states given the current context. CPC's contrastive objective provides a lower bound on this quantity, avoiding the intractability of direct MI computation in high-dimensional spaces. This principle distinguishes CPC from reconstruction-based methods like autoencoders: instead of modeling irrelevant pixel-level details, CPC captures slow features and high-level semantics that persist across time, making it effective for phoneme recognition and speaker identification.

The quality of CPC representations depends critically on negative sample selection. The original formulation uses in-batch negatives, where other timesteps from the same sequence or other sequences in the mini-batch serve as distractors. For audio and speech tasks, negatives can be drawn from:

• Same sequence, different timesteps: Forces the model to distinguish temporal progression
• Different sequences: Encourages speaker-invariant or content-discriminative features
• Hard negative mining: Selecting negatives with high similarity to the anchor sharpens decision boundaries

Poor negative sampling leads to representation collapse, where the encoder maps all inputs to a trivial constant vector.

The broader family of objectives that CPC belongs to. Contrastive loss functions operate on pairs or triplets of samples, pulling positive pairs (semantically related) together in embedding space while pushing negative pairs (unrelated) apart beyond a specified margin. Unlike triplet loss, which requires careful triplet mining, CPC's InfoNCE formulation naturally handles multiple negatives per positive. The temperature parameter controls the concentration of the similarity distribution: lower temperatures sharpen the penalty on hard negatives, improving feature discrimination but risking training instability.

A catastrophic failure mode where the encoder learns to output a constant vector regardless of input, achieving zero contrastive loss trivially. CPC avoids collapse through its autoregressive constraint: the context model must predict future latents, preventing the trivial solution where all representations are identical. Other frameworks address collapse differently: BYOL uses a predictor MLP and stop-gradient, Barlow Twins decorrelates embedding dimensions, and VICReg explicitly regularizes variance. Understanding collapse prevention mechanisms is essential when adapting CPC to non-sequential domains like images.