Self-Supervised Learning (SSL)

A pretext task is an auxiliary, automatically generated learning objective used to train a model on unlabeled data. The model learns useful representations by solving a surrogate problem, with the 'labels' derived from the data itself.

Common examples include:

• Inpainting/Masked Prediction: Reconstructing randomly masked portions of an image (e.g., MAE) or text (e.g., BERT).
• Rotation Prediction: Classifying the applied rotation (0°, 90°, 180°, 270°) of an input image.
• Jigsaw Puzzle Solving: Rearranging shuffled image patches into their original configuration.
• Colorization: Predicting the color channels of a grayscale image.

The key is that solving these tasks forces the model to learn semantic features about the structure and content of the data, which transfer well to downstream supervised tasks.

Contrastive learning trains an encoder to produce similar representations (embeddings) for semantically related data points (a positive pair) and dissimilar representations for unrelated points (negatives). It is defined by three core elements:

• Data Augmentation Pipeline: Creates different 'views' of the same instance (e.g., cropped, color-jittered versions of an image) to form positive pairs.
• Projection Head: A small neural network (often an MLP) that maps the backbone encoder's output to a lower-dimensional space where the contrastive loss is applied.
• Contrastive Loss Function: Such as InfoNCE Loss, which pulls positive pairs together and pushes negatives apart in the embedding space.

Frameworks like SimCLR and MoCo popularized this approach, which optimizes for alignment (similarity of positives) and uniformity (even distribution of all embeddings).

Non-contrastive learning is a category of SSL methods that learn effective representations without directly comparing negative samples. These methods avoid the computational and theoretical challenges of maintaining large sets of negatives.

Core techniques include:

• Architectural Asymmetry: Using a predictor head on one branch and a stop-gradient operation, as in SimSiam and BYOL, to prevent collapse.
• Redundancy Reduction: Enforcing decorrelation between embedding dimensions, as in Barlow Twins and VICReg, to learn informative, non-collapsed features.
• Self-Distillation: Using a momentum-updated teacher network to generate targets for a student network, as in DINO and BYOL.

These methods prove that effective SSL is possible through careful regularization and architectural design, not solely through contrastive objectives.

Online clustering generates supervisory signals by assigning data points to cluster centroids on-the-fly during training. The model learns by predicting the cluster assignment of one augmented view from another view of the same image.

SwAV (Swapping Assignments between Views) is the canonical example:

• It computes 'codes' (soft cluster assignments) for different augmented views of an image using a set of trainable prototype vectors.
• The learning objective is to swap these predictions: the representation from one view should predict the code of the other view.
• This creates a cross-view prediction task that avoids the need for pre-computed labels or an external memory bank of negatives.

This mechanism is highly scalable and efficient, enabling learning from large, uncurated datasets by creating discrete, online-generated targets.

A momentum encoder is a slowly updated copy of the main (online) encoder, used to provide stable, consistent targets for representation learning. It is a key component in frameworks like MoCo and BYOL.

How it works:

• The momentum encoder's parameters are not trained via gradient descent. Instead, they are updated via an exponential moving average (EMA) of the online encoder's parameters.
• This creates a slowly evolving 'teacher' network that produces targets for the 'student' online network to predict.
• The update rule is: θₖ ← m * θₖ + (1 - m) * θ_q, where θₖ are the momentum encoder weights, θ_q are the online encoder weights, and m is a momentum coefficient (e.g., 0.999).

This mechanism prevents rapid target drift, stabilizes training, and is crucial for avoiding representation collapse in non-contrastive methods.

Masked reconstruction, popularized by Masked Autoencoders (MAE), is a generative pretext task where a model learns to reconstruct the original input from a partially masked version. This forces the model to develop a comprehensive understanding of data structure and context.

Key characteristics:

• A high proportion of the input (e.g., 75% of image patches) is randomly masked.
• The encoder operates only on the visible patches, creating a latent representation.
• A lightweight decoder reconstructs the full original input from the latent representation and mask tokens.
• The loss (e.g., Mean Squared Error) is computed only on the masked patches.

This asymmetric design makes training efficient while compelling the model to learn robust, semantic features by predicting missing content based on global context. It has become a dominant paradigm in vision and language.

Contrastive learning trains a model to produce similar embeddings for semantically related data points (positive pairs) and dissimilar embeddings for unrelated points (negative pairs). This is achieved by maximizing agreement between different augmented views of the same data instance.

• Core Mechanism: Uses a contrastive loss function like InfoNCE to pull positive pairs together and push negatives apart in the embedding space.
• Key Property: Enforces invariance to predefined data augmentations, learning representations that are robust to noise and irrelevant transformations.
• Examples: SimCLR, MoCo, and NT-Xent loss are seminal contrastive frameworks.

Non-contrastive learning methods learn useful representations without directly comparing negative samples. They avoid the computational and theoretical challenges of maintaining large negative sample sets.

• Core Mechanism: Relies on architectural asymmetry (e.g., predictor networks, stop-gradient operations) and regularization constraints to prevent representation collapse.
• Key Methods: BYOL uses a momentum teacher network. Barlow Twins minimizes redundancy between embedding dimensions. VICReg enforces variance, invariance, and covariance constraints.
• Advantage: Often requires smaller batch sizes and is less sensitive to the choice of negative sampling strategy.

Generative pretext tasks train a model to reconstruct or predict parts of its input, forcing it to learn a rich, internal model of the data's structure.

• Core Mechanism: The model is trained with a reconstruction loss, such as mean squared error or cross-entropy, on a corrupted or partial input.
• Primary Examples:
  • Masked Autoencoders (MAE): Randomly mask a high proportion of input patches (e.g., 75% of an image) and train a model to reconstruct the missing parts.
  • Denoising Autoencoders: Reconstruct the original input from a noised version.
• Outcome: The model learns high-level features and dependencies within the data, which transfer well to downstream discriminative tasks.

Clustering-based methods generate supervisory signals by assigning data points to clusters or prototypes in an online manner during training.

• Core Mechanism: Alternates between assigning embeddings to cluster centroids and updating the model parameters to produce embeddings that match these assignments.
• Key Method: SwAV (Swapping Assignments between Views) computes a code (cluster assignment) for one augmented view of an image and tries to predict it from another view.
• Advantages:
  • Eliminates the need for explicit negative pairs.
  • The clustering acts as an online pseudo-labeling mechanism, creating a discrete learning target.
  • Naturally scales to large datasets.

Distillation-based methods leverage a self-supervised knowledge distillation setup, where a student network learns to match the output of a teacher network, with both networks processing different views of the same data.

• Core Mechanism: Uses a teacher-student architecture where the teacher's parameters are an exponential moving average (EMA) of the student's. The student is trained to predict the teacher's output.
• Key Methods:
  • DINO: Uses a centering and sharpening operation on the teacher's output to encourage the emergence of semantic features without labels.
  • SimSiam: Employs a simple siamese network with a stop-gradient operation on one branch and a predictor head.
• Outcome: The student learns a consistent and semantically meaningful representation space guided by the more stable teacher.

Cross-modal alignment creates supervisory signals by leveraging the natural correspondence between different modalities of the same data instance (e.g., an image and its caption, a video and its audio).

• Core Mechanism: Trains a model to maximize the similarity between embeddings of paired multimodal data while minimizing similarity for unpaired data.
• Training Objective: Often uses a contrastive loss (e.g., CLIP's contrastive loss) to align representations across modalities in a shared embedding space.
• Applications:
  • Foundation for models like CLIP (Contrastive Language-Image Pre-training).
  • Critical for continual learning in multimodal data streams, where correspondence between modalities provides a powerful, free supervisory signal.

Contrastive learning is a dominant SSL technique that trains a model to produce similar representations for semantically related data points (positive pairs) and dissimilar representations for unrelated points (negative pairs).

• Core mechanism: It defines a pretext task of discriminating between samples.
• Key objective: To learn an embedding space where similarity reflects semantic relatedness.
• Example: In computer vision, two different augmented views of the same image form a positive pair, while views from different images form negative pairs.

A pretext task is an auxiliary, automatically generated learning objective used to train a model on unlabeled data. The model's performance on this synthetic task is not the end goal; rather, solving it forces the model to learn useful, general-purpose representations.

• Common examples: Predicting the rotation angle of an image, solving a jigsaw puzzle of image patches, reconstructing masked portions of the input (as in Masked Autoencoders), or predicting the relative position of image patches.
• Purpose: Provides the 'self-supervision' signal that replaces human-annotated labels during pre-training.

Non-contrastive learning is a category of SSL methods that learn useful representations without directly comparing negative samples. These methods avoid the computational and theoretical complexities of maintaining and comparing large sets of negatives.

• Core principle: Uses architectural asymmetry (e.g., predictor networks, stop-gradient operations) and regularization to prevent representational collapse.
• Key methods: BYOL (Bootstrap Your Own Latent) and Barlow Twins are seminal examples. They enforce invariance to augmentations and reduce redundancy between embedding dimensions, respectively.
• Advantage: Often requires smaller batch sizes than contrastive methods.

The linear evaluation protocol is the standard benchmark for assessing the quality of representations learned via SSL. It measures how well the learned features generalize to downstream tasks.

• Procedure: The pretrained encoder's weights are frozen. A single linear classification layer (a projection head is typically removed) is then trained on top of the frozen features using a fully labeled dataset (e.g., ImageNet).
• Purpose: Isolates the quality of the representations from further fine-tuning, providing a clean measure of the encoder's feature extraction capability.
• Alternative: k-NN evaluation offers a non-parametric, training-free assessment using the same frozen features.

A momentum encoder is a key architectural component in frameworks like MoCo (Momentum Contrast). It is a slowly updated copy of the main online encoder, designed to provide consistent, stable targets for contrastive learning.

• Update rule: Its parameters are an exponential moving average (EMA) of the online encoder's parameters, updated with a high momentum value (e.g., 0.999).
• Function: Maintains a consistent feature dictionary for negative samples, preventing rapid fluctuation of the target representations that the online network is trying to match.
• Benefit: Enables the use of a large, consistent set of negative samples without requiring a massive batch size.

Continual self-supervised learning extends SSL to non-stationary data streams, where a model must learn new representations from an unlabeled data sequence without catastrophically forgetting previously learned ones. This is a core challenge within the Continuous Model Learning Systems pillar.

• Key problem: Mitigating feature drift, where the model's internal representations degrade for past data as it adapts to new data.
• Techniques: Often incorporates experience replay mechanisms with stored past data or uses regularization methods to constrain weight updates.
• Goal: To enable lifelong, adaptive pre-training of foundation models on ever-evolving data sources.