Contrastive Learning

The engine of contrastive learning is the creation of positive pairs—two views of the same underlying sample. In multi-omics, this is achieved through:

• Data augmentation: Applying stochastic transformations (e.g., masking genes, adding Gaussian noise) to the same single-cell transcriptomic profile.
• Natural pairing: Using different omics modalities (e.g., mRNA and protein expression from CITE-seq) measured simultaneously in the same cell as a positive pair.
• Context-based: Defining cells from the same tissue region or developmental time point as semantically similar.

The model is trained to map these positive pairs to nearby points in the embedding space, learning invariance to the applied augmentations.

Equally critical is the selection of negative samples—data points the model must push away in the embedding space. Key strategies include:

• In-batch negatives: All other samples in a training mini-batch are treated as negatives, as done in SimCLR. This is computationally efficient but requires large batch sizes.
• Memory bank: A running queue of representations from recent batches is maintained, decoupling negative sample count from batch size (MoCo).
• Hard negative mining: Actively selecting negatives that the model currently confuses with the anchor, forcing the learning of finer-grained distinctions between closely related cell types. Poor negative sampling can lead to class collision, where a true biological match is incorrectly pushed away.

The standard training objective is the InfoNCE (Noise Contrastive Estimation) loss, a categorical cross-entropy that treats the positive pair as the correct class among all negatives. For an anchor z_i and its positive z_j:

• The loss computes the cosine similarity between z_i and all other representations.
• A temperature parameter (τ) controls the concentration of the distribution. Lower τ sharpens the distribution, focusing the model on hard negatives.
• Minimizing InfoNCE maximizes a lower bound on mutual information between the learned representations of positive pairs. This loss is the mathematical core of models like SimCLR, MoCo, and CLIP.

A critical architectural insight is the use of a non-linear projection head—a small multi-layer perceptron (MLP) placed on top of the encoder backbone. Key findings:

• The contrastive loss is applied to the output of the projection head, not the encoder's final representation directly.
• After pre-training, the projection head is discarded, and the encoder's representation is used for downstream tasks.
• The projection head filters out information irrelevant to the contrastive task, allowing the encoder to retain richer, more generalizable features. This design prevents the loss from overfitting to the specific augmentations used during training.

Contrastive learning excels at cross-modal alignment—mapping data from different sources into a shared embedding space without paired labels. In multi-omics:

• CLIP-inspired models: A transcriptomic encoder and a proteomic encoder are trained jointly. A positive pair is the mRNA and protein profile of the same cell; all other combinations are negatives.
• Translation: Once aligned, a model can translate between modalities—predicting protein expression from mRNA alone, or vice versa.
• Zero-shot retrieval: A query in one modality (e.g., a gene expression pattern) can retrieve the most similar profiles in another modality (e.g., a cellular morphology image) by searching the shared embedding space. This enables the integration of modalities that lack one-to-one sample correspondence.

To maintain a stable and consistent representation space, architectures like MoCo (Momentum Contrast) use a dual-encoder design:

• A query encoder is updated by standard backpropagation.
• A key encoder is updated as an exponential moving average (EMA) of the query encoder's weights: θ_k ← m·θ_k + (1-m)·θ_q, where m is a momentum coefficient close to 1 (e.g., 0.999).
• The slowly evolving key encoder produces consistent representations for the memory bank, preventing rapid changes from creating spurious negatives. This EMA strategy is now a standard component in many self-supervised learning frameworks, including BYOL and DINO.

The computational process of combining data from different 'omics' layers—such as genomics, transcriptomics, proteomics, and metabolomics—into a unified view. Contrastive learning is a primary engine for this, aligning cells or features from different modalities into a shared latent space where similar biological states cluster together regardless of their data origin.

• Enables cross-modal prediction (e.g., predicting protein abundance from mRNA)
• Resolves cellular heterogeneity that single-modality analysis misses
• Foundational for constructing comprehensive single-cell atlases

A large, pre-trained neural network—such as scGPT or Geneformer—trained on massive corpora of single-cell transcriptomic data using self-supervised objectives. Contrastive learning is central to these models, teaching them to generate general-purpose cellular representations that distinguish between cell types and states.

• Pre-trained on tens of millions of cells
• Fine-tuned for downstream tasks like cell-type annotation or perturbation prediction
• Leverages the attention mechanism to learn context-specific gene interactions

A mathematical framework for finding the most efficient mapping between two probability distributions. In single-cell biology, it is used to align cells from different batches, time points, or omics modalities by minimizing a cost function based on feature similarity. It is often paired with contrastive objectives to enforce a smooth, geometrically meaningful alignment.

• Gromov-Wasserstein distance aligns datasets with different feature spaces
• Used in spatial transcriptomics to map cells to tissue locations
• Provides a principled alternative to simple nearest-neighbor matching

A computational technique to remove non-biological, technical variation introduced by different experimental handling, sequencing platforms, or processing times. Contrastive learning approaches explicitly train models to be invariant to batch identity while preserving true biological variation, ensuring integrated data reflects biology rather than technical artifacts.

• Critical for merging datasets from multiple laboratories
• Contrastive methods like scVI and Harmony outperform older methods
• Prevents spurious clusters driven by technical rather than biological differences

Cellular Indexing of Transcriptomes and Epitopes by Sequencing is a multi-omics technology that simultaneously profiles the whole transcriptome and a panel of cell-surface proteins from the same single cell. It provides a natural paired dataset for training and validating contrastive learning models that learn joint representations of RNA and protein modalities.

• Uses oligonucleotide-conjugated antibodies to tag surface proteins
• Generates paired modalities ideal for benchmarking alignment algorithms
• Enables direct correlation of mRNA levels with actual protein abundance

A computational method that predicts the future transcriptional state of individual cells by modeling the ratio of unspliced to spliced mRNA. Contrastive learning can be applied to align the current and predicted future states of a cell, learning representations that encode a directional vector of cellular differentiation and developmental trajectories.

• Infers future cell state without time-series experiments
• Contrastive objectives can align current and predicted states
• Essential for reconstructing trajectory inference and lineage decisions