Contrastive Predictive Coding (CPC)

CPC uses a noise-contrastive estimation loss called InfoNCE to train the encoder. The model must distinguish a true future latent from a set of negative samples drawn from the proposal distribution.

• Maximizes a lower bound on mutual information between context and future latents
• The loss is computed as a categorical cross-entropy over the positive and N-1 negative samples
• A higher N (more negatives) improves the tightness of the mutual information bound
• The log-bilinear density ratio scores compatibility between context vectors and target embeddings

CPC employs a unidirectional recurrent or masked transformer that summarizes all past latent observations into a single compact context vector. This context is then used to predict future latents multiple steps ahead.

• Typically implemented with a GRU or causal transformer decoder
• The context vector c_t aggregates information from z_≤t
• Enables prediction of latents at multiple future time offsets (e.g., z_{t+1}, z_{t+2}, z_{t+4})
• The architecture enforces a bottleneck that discards local noise and retains global structure

By predicting across long temporal or spatial horizons, CPC forces the encoder to learn slowly varying features that are shared across distant patches of the sequence. High-frequency, unpredictable noise is naturally discarded.

• The prediction horizon is a critical hyperparameter controlling abstraction level
• Short horizons capture local texture; long horizons capture global semantics
• In genomics, this extracts regulatory motifs that span hundreds of base pairs
• The mutual information maximization principle is agnostic to the data modality

When applied to DNA sequences, CPC learns embeddings that capture functional regulatory syntax without requiring labeled data. The model predicts the latent representation of a downstream genomic region from the context of an upstream region.

• Tokenized nucleotides or k-mers are encoded into latent vectors z_t
• The context network summarizes a promoter or enhancer region
• The model predicts the embedding of a distal transcription factor binding site or exon
• Learned representations transfer effectively to variant effect prediction and chromatin state classification

CPC is a foundational example of the contrastive pre-training paradigm that preceded and influenced models like SimCLR and MoCo. It demonstrated that self-supervised learning could rival supervised pre-training on downstream tasks.

• Published by van den Oord et al. at DeepMind in 2018
• Originally demonstrated on speech, images, text, and reinforcement learning environments
• The strided cropping strategy for images creates spatial patches analogous to temporal windows
• Established the principle that predicting in latent space is superior to predicting in pixel or token space

The choice of negative samples critically impacts representation quality. CPC samples negatives from the same sequence or batch, forcing the model to discriminate between genuine long-range dependencies and spurious correlations.

• Negatives can be drawn from other positions in the same sequence or from other sequences in the batch
• Hard negative mining selects negatives that are most confusable with the positive
• In genomics, negatives from the same chromosome prevent the model from learning trivial GC-content biases
• The strategy implicitly defines what the representation should be invariant to