Contrastive Loss

Contrastive loss is classically implemented using a Siamese network—two identical subnetworks with shared weights that process a pair of inputs in parallel. Each branch produces an embedding vector, and the loss is computed on the Euclidean distance between these outputs. The architecture enforces weight symmetry, ensuring that the same genomic sequence processed through either branch yields identical representations. During training on orthologous promoter pairs, both branches receive different sequences, and the shared weights learn a unified mapping function that generalizes across the input distribution.

The margin (often denoted as m) is a critical hyperparameter that defines the radius of influence for negative pairs. Once the distance between a dissimilar pair exceeds the margin, the loss contribution becomes zero—the model stops pushing them further apart. This prevents the embedding space from expanding indefinitely and focuses optimization on hard negatives that lie within the margin boundary. In genomic applications, setting the margin too low collapses distinct regulatory elements into overlapping clusters, while too high a margin wastes capacity on separating already-distant sequences like promoters from intergenic deserts.

Training requires explicit positive pairs (similar genomic regions) and negative pairs (dissimilar regions). Positive pairs are constructed from:

• Orthologous promoters across species (e.g., human BRCA1 promoter paired with mouse Brca1 promoter)
• Augmented views of the same sequence via reverse complement or random masking Negative pairs are sampled from:
• Randomly selected genomic loci from different chromosomes
• Promoters of functionally unrelated genes
• Hard negative mining: sequences with similar GC content but different regulatory function

The standard contrastive loss is defined as:

L = (1 - Y) × ½(D)² + Y × ½{max(0, m - D)}²

Where:

• Y = 0 for positive pairs, 1 for negative pairs
• D = Euclidean distance between the two embedding vectors
• m = margin hyperparameter

For positive pairs (Y=0), only the first term activates, minimizing the squared distance. For negative pairs (Y=1), the second term penalizes distances smaller than the margin, pushing embeddings apart until D ≥ m.

While contrastive loss operates on pairs of samples, triplet loss extends this to triplets (anchor, positive, negative). Key differences:

• Contrastive loss uses absolute distances; triplet loss enforces relative distances: D(anchor, positive) + margin < D(anchor, negative)
• Contrastive loss is symmetric in its treatment of the pair; triplet loss is asymmetric around the anchor
• For genomic sequence embedding, contrastive loss is preferred when the notion of similarity is bidirectional (e.g., orthologous regions), while triplet loss suits directional queries (e.g., retrieving enhancers similar to a query promoter)

A well-trained contrastive loss produces an embedding space with specific geometric properties:

• Semantic organization: Functionally similar regulatory elements cluster together regardless of sequence identity
• Smooth interpolation: Linear interpolation between two promoter embeddings passes through embeddings of promoters with intermediate activity levels
• Distance calibration: Euclidean distance correlates with functional dissimilarity, enabling threshold-based retrieval
• Strand invariance: When trained with reverse complement augmentation, the forward and reverse strand of the same locus map to identical coordinates

A precursor to contrastive loss that uses anchor, positive, and negative triplets. The objective ensures the distance between the anchor and a positive example (e.g., an orthologous promoter) is less than the distance to a negative example (e.g., a random intergenic region) by a defined margin.

• Hard Negative Mining: Critical for convergence; selects negatives that are close to the anchor but semantically different.
• Use Case: Learning fine-grained functional similarity between regulatory elements across species.

The Normalized Temperature-scaled Cross Entropy Loss is the standard objective used in SimCLR. It operates on a batch of augmented pairs, treating the other 2(N-1) augmented examples as negatives.

• Temperature Parameter: Controls the concentration of the distribution; lower values sharpen the penalty on hard negatives.
• Genomic Adaptation: Applied to DNA sequence augmentations like random masking or reverse complementing to learn strand-invariant embeddings.

A self-supervised method that maximizes mutual information between latent representations of past and future sequence patches. A powerful autoregressive model extracts slow features, while a density ratio estimator scores the agreement.

• Genomic Application: Predicts downstream chromatin states or epigenetic marks from upstream DNA sequence context.
• Key Advantage: Learns high-level representations that discard low-level noise, capturing long-range regulatory syntax.

The fundamental metric for measuring vector alignment in a contrastive latent space. It computes the cosine of the angle between two L2-normalized embedding vectors, yielding a score from -1 to 1.

• Contrastive Role: The scoring function used to determine if two genomic regions (e.g., enhancer-promoter pairs) are functionally linked.
• Efficiency: Highly optimized for approximate nearest neighbor (ANN) search in vector databases storing genomic embeddings.

A simple framework for contrastive learning of visual representations, adapted for genomics. It creates two augmented views of the same DNA sequence, passes them through an encoder and a non-linear projection head, and maximizes agreement using NT-Xent loss.

• Key Insight: The projection head discards augmentation-specific information before the contrastive step.
• Genomic Augmentations: Includes random k-mer masking, reverse complementing, and localized shuffling of sequence windows.

A sampling strategy essential for preventing representational collapse in contrastive loss. Instead of random negatives, the model is trained on negatives that are difficult to distinguish from positives.

• Genomic Strategy: Selecting pseudo-genes or non-functional paralogs as negatives for a functional gene anchor.
• Benefit: Sharpens decision boundaries in the latent space, forcing the model to learn discriminative regulatory features rather than superficial sequence similarity.