SimCLR

SimCLR generates two correlated views of the same genomic sequence through a composition of domain-specific augmentations. These include random k-mer masking, nucleotide substitution with IUPAC ambiguity codes, reverse complement flipping, and random cropping of genomic windows. The model is trained to recognize that these perturbed versions represent the same underlying biological locus, forcing the encoder to learn features invariant to sequencing noise and natural variation. This augmentation pipeline is critical for preventing the model from learning trivial shortcuts.

A small multi-layer perceptron (MLP) is attached to the top of the genomic encoder during training. This projection head maps the encoder's output embeddings to a lower-dimensional latent space where the contrastive loss is applied. Crucially, this head is discarded after pre-training. Research shows that the projection head filters out augmentation-specific information, allowing the encoder's representations to retain more generalizable biological features. The final embeddings used for downstream tasks come directly from the encoder, not the projection head.

SimCLR uses the Normalized Temperature-scaled Cross Entropy (NT-Xent) loss. Within a mini-batch of N genomic sequences, each augmented view is treated as an anchor. The positive pair is the other augmented view of the same sequence, while the remaining 2(N-1) augmented views are treated as negative examples. The loss function computes the cosine similarity between all pairs, scales them by a temperature parameter, and applies a softmax to maximize agreement between positive pairs while repelling negatives. This effectively organizes the embedding space by biological function.

SimCLR's performance is highly dependent on large batch sizes, typically ranging from 256 to 4096 sequences. A large batch size provides more negative examples per iteration, which is essential for the contrastive loss to learn meaningful distinctions between genomic regions. For genomic applications, this requires significant GPU memory and distributed training infrastructure. Techniques like gradient accumulation and mixed-precision training are often employed to simulate large batches on limited hardware, though this may impact the quality of the learned representations.

A key augmentation for genomics is the reverse complement transformation. DNA is double-stranded, and a regulatory element functions identically on both strands. By including reverse complement as a standard augmentation, SimCLR learns embeddings that are inherently strand-agnostic. The encoder maps a sequence and its reverse complement to nearly identical points in the latent space, eliminating the need for strand-specific post-processing. This property is critical for tasks like transcription factor binding site prediction, where the orientation of the motif is biologically irrelevant.

Once pre-trained on a large, unlabeled genomic corpus, the frozen encoder can be used to generate fixed feature vectors for supervised tasks with limited labeled data. Common downstream applications include:

• Promoter strength prediction: Using embeddings to regress gene expression levels
• Variant effect scoring: Comparing embeddings of reference and mutated sequences
• Chromatin state classification: Fine-tuning a linear classifier on top of frozen embeddings This approach achieves state-of-the-art performance even when only hundreds of labeled examples are available.

A self-supervised method that trains an encoder to extract slowly varying features by maximizing mutual information between latent representations of temporally or spatially distant sequence patches. In genomics, CPC learns embeddings that capture long-range regulatory syntax by predicting future k-mer representations from past context, making it effective for identifying distal enhancer-promoter interactions without labeled data.

The mathematical objective function that pulls embeddings of positive pairs (augmented views of the same sequence) closer in latent space while pushing negative pairs (views from different sequences) apart. In SimCLR, this is implemented as the NT-Xent loss (normalized temperature-scaled cross-entropy), which operates on the cosine similarity matrix of a mini-batch to learn a semantically organized manifold of genomic elements.

A metric measuring the cosine of the angle between two embedding vectors, ranging from -1 (opposite) to 1 (identical). In SimCLR's genomic applications, cosine similarity quantifies the functional relatedness of regulatory elements in the latent space:

• Orthologous promoters from different species cluster with high similarity
• Tissue-specific enhancers separate along axes of cell-type identity
• The metric enables efficient nearest-neighbor retrieval of functionally analogous sequences

A domain-specific data augmentation that doubles the training dataset by presenting both the forward strand and its reverse complement to the model. This enforces strand-invariance in the learned embeddings—a critical inductive bias for genomic models, since regulatory proteins bind DNA regardless of strand orientation. In SimCLR, this serves as one of the two augmented views for each input sequence.

An alternative self-supervised architecture that masks a high proportion of input DNA patches (typically 75-80%), trains an asymmetric encoder-decoder on only visible tokens, and reconstructs the masked nucleotides. Unlike SimCLR's contrastive approach, MAE learns by generative reconstruction rather than instance discrimination, producing highly efficient latent representations that excel at fine-grained nucleotide-level prediction tasks.

A small non-linear multi-layer perceptron (typically 2-3 layers) appended to the encoder during SimCLR training. The contrastive loss is applied to the output of this head, not the encoder's representation directly. After pre-training, the projection head is discarded, and the encoder's penultimate layer serves as the downstream embedding. This architectural trick prevents the loss from discarding information useful for transfer learning.