Contrastive Learning

The fundamental dynamic of contrastive learning operates on a simple geometric principle:

• Positive Pairs: Two augmented views of the same input (e.g., a waveform with different noise additions) are pulled together in the embedding space.
• Negative Pairs: Views from different inputs are pushed apart.
• Noise-Contrastive Estimation: The model is trained to identify the positive sample among a set of negative distractors, learning representations invariant to irrelevant transformations while preserving identity-discriminative information.

The dominant objective function in modern contrastive learning frameworks:

• Formulates the task as a categorical cross-entropy problem over a batch, where the correct pair is classified among N candidates.
• The loss minimizes the negative log-likelihood of identifying the positive pair, effectively maximizing mutual information between different views of the same underlying signal.
• A temperature parameter (τ) controls the concentration of the distribution, sharpening or softening the penalty on hard negative samples that lie close to the anchor in the embedding space.

The choice of augmentations defines the invariances the model learns, making it the most critical design decision in contrastive learning:

• Signal-Specific Augmentations: For RF fingerprinting, augmentations include additive white Gaussian noise, frequency offset, phase rotation, and simulated multipath fading.
• Invariance Induction: By treating augmented versions as the same identity, the model learns to ignore channel effects and focus on hardware-intrinsic impairments.
• Augmentation Stacking: Composing multiple transformations forces the encoder to capture high-level semantic features rather than low-level signal statistics.

Not all negative samples contribute equally to learning. Hard negative mining identifies the most informative contrasts:

• Hard Negatives: Samples from different classes that are mapped close to the anchor in the current embedding space, creating a challenging discrimination task.
• Semi-Hard Negatives: Negatives that are farther from the anchor than the positive but still within a margin, providing a useful gradient signal without destabilizing training.
• In open set emitter recognition, hard negatives often represent transmitters of the same model with nearly identical hardware impairments, forcing the model to learn subtle discriminative features.

Popularized by frameworks like MoCo, the momentum encoder provides a stable target representation:

• Slow-Moving Average: A secondary encoder whose weights are updated via exponential moving average of the primary encoder's weights, preventing representation collapse.
• Dynamic Dictionary: The momentum encoder builds a large, consistent queue of negative sample representations, decoupling the dictionary size from the batch size.
• This architecture is particularly valuable for RF applications where large batches of high-dimensional IQ samples are computationally prohibitive, yet a diverse negative set is essential for discriminative learning.

A small multi-layer perceptron appended to the encoder during training, but discarded for downstream tasks:

• Dimensionality Expansion: The projection head maps representations to a higher-dimensional space where contrastive loss is applied, preventing the loss from discarding useful information.
• Collapse Prevention: Without careful regularization, contrastive learning can suffer from dimensional collapse, where the embedding space degenerates to a low-rank subspace. Techniques like variance regularization and covariance regularization explicitly penalize this collapse.
• After training, the frozen encoder outputs are used directly for emitter classification or open set rejection.

The foundational noise-contrastive estimation objective that treats the positive pair as the signal and all other samples in the batch as noise. It maximizes the mutual information between different views of the same data point by computing a categorical cross-entropy over similarity scores.

• Formulated as a log-softmax over cosine similarities
• Temperature parameter τ controls concentration of the distribution
• Lower τ creates harder negatives and sharper decision boundaries
• Used in SimCLR, CPC, and MoCo frameworks

A classic metric learning objective that operates on triplets of (anchor, positive, negative) samples. It enforces that the distance between the anchor and positive is smaller than the distance between the anchor and negative by at least a specified margin α.

• Loss = max(d(anchor, positive) - d(anchor, negative) + margin, 0)
• Requires careful hard negative mining to avoid collapsed representations
• Margin α defines the minimum enforced separation in embedding space
• Widely used in FaceNet for facial recognition tasks

The Normalized Temperature-scaled Cross Entropy loss used in the SimCLR framework. It computes pairwise similarity between all augmented views in a batch using cosine similarity on L2-normalized embeddings.

• Operates on positive pairs derived from data augmentation
• All other 2(N-1) samples in the batch serve as negatives
• Symmetric formulation: both (i,j) and (j,i) pairs contribute to loss
• Scales efficiently with large batch sizes for more negative examples

An architectural pattern that maintains a dynamic dictionary of encoded representations using a momentum-updated encoder rather than relying solely on the current mini-batch for negative samples.

• Key encoder θ_k updates slowly: θ_k ← m·θ_k + (1-m)·θ_q
• Momentum coefficient m typically set to 0.999 for stability
• Queue stores thousands of encoded keys for consistent negatives
• Decouples dictionary size from batch size, enabling richer contrast

The strategic selection of negative samples that are difficult to distinguish from the anchor, forcing the model to learn more discriminative features. Random negatives often provide trivial loss signals.

• Hard negatives have high similarity to the anchor but belong to different classes
• Semi-hard negatives lie within the margin band of triplet loss
• Improves convergence speed and final embedding quality
• Critical for open set recognition where fine-grained distinctions matter

A Siamese network variant that achieves competitive representations without negative pairs, momentum encoders, or large batches. It uses a stop-gradient operation on one branch to prevent representational collapse.

• Predictor MLP on one branch maps to the other branch's output
• Stop-gradient breaks symmetry: no gradients flow through target encoder
• Relies on the inductive bias of the architecture rather than contrast
• Demonstrates that negative samples are not strictly necessary