Self-Supervised Learning

A technique that learns representations by training a model to distinguish between similar (positive) and dissimilar (negative) data pairs. The objective is to pull the embeddings of positive pairs (e.g., different augmentations of the same image) closer together in the latent space while pushing negative pairs apart.

• Key Mechanism: Uses a contrastive loss function, such as InfoNCE or NT-Xent.
• Core Challenge: Requires careful construction of positive/negative pairs. Avoiding collapsed representations (where all outputs are identical) is critical.
• Examples: SimCLR and MoCo for computer vision; Sentence-BERT for natural language.

A technique where a model is trained to reconstruct or generate the original input data from a corrupted or partial version. The model learns a rich internal representation by mastering the data distribution.

• Core Tasks: Includes masked language modeling (e.g., BERT), image inpainting, and denoising autoencoders.
• Mechanism: The model is presented with an input where parts are removed (masked) or corrupted with noise, and it must predict the missing original content.
• Outcome: The model develops a world model—an understanding of data structure and context—essential for downstream discriminative tasks.

A framework inspired by neuroscience where a model learns by predicting future or missing information in a temporal or spatial sequence. It emphasizes learning the latent state dynamics of the environment.

• Temporal Context: Predicting the next frame in a video or the next word in a sequence (e.g., GPT's autoregressive training).
• Spatial Context: Predicting neighboring patches in an image or surrounding words in a sentence.
• Objective: Minimizes the prediction error between the model's forecast and the actual observed data, forcing it to learn causal and correlational structures.

Techniques that generate labels by clustering the data representations themselves, then use these cluster assignments as pseudo-labels for a classification task. This creates an iterative self-labeling process.

• Process: 1. Learn initial features via a pretext task. 2. Cluster the feature embeddings. 3. Use cluster IDs as targets to re-train the network, improving the features. 4. Repeat.
• Key Benefit: Avoids the need for explicit negative samples required in contrastive learning.
• Examples: DeepCluster and SwAV for image data. These methods are closely related to learning disentangled representations.

A family of techniques that learn representations by having two neural network branches (online and target) agree on representations of different augmented views of the same input, without using negative pairs.

• Core Idea: The online network is trained to predict the output of a slowly evolving target network (or a stopped-gradient version of itself).
• Avoiding Collapse: Prevents representation collapse through architectural tricks like stop-gradient, momentum encoders, or predictor networks.
• Significance: Demonstrates that contrastive learning's negative samples are not strictly necessary, simplifying the training pipeline.

A principle from information theory applied to SSL, aiming to learn representations where each component is statistically independent, thereby removing redundant information from the input.

• Objective: Maximize the information content of the learned latent space by making features uncorrelated and non-redundant.
• Implementation: Often achieved via whitening operations in the embedding space or through specific loss functions that minimize the mutual information between feature dimensions.
• Connection: This principle is foundational to learning disentangled representations and is a hypothesized goal of the brain's sensory processing pathways.

A dominant self-supervised technique where a model learns representations by contrasting similar and dissimilar data points. The core objective is to learn an embedding space where semantically similar samples (positive pairs) are pulled together, while dissimilar ones (negative pairs) are pushed apart.

• Key Mechanism: Uses a contrastive loss function, like InfoNCE, to maximize agreement between differently augmented views of the same data instance.
• Common Architecture: Often employs a Siamese network with a shared encoder.
• Example: In computer vision, SimCLR and MoCo create positive pairs by applying random crops, color jitter, and blur to the same image, treating all other images in the batch as negatives.

A type of model that learns the underlying probability distribution of the training data, enabling it to generate new, plausible samples. While self-supervised learning is a broad training paradigm, many generative models are trained using self-supervised objectives.

• Core Objective: Learn ( p(x) ), the distribution of the data.
• Self-Supervised Tasks: Common pretext tasks include masked token prediction (e.g., BERT, GPT) and image inpainting.
• Contrast with Discriminative Models: Discriminative models learn ( p(y|x) ) (the probability of a label given data). Generative models learn the data itself, which is a more general and often more difficult task.

The overarching field concerned with automatically discovering informative, compressed feature representations from raw data. Self-supervised learning is a primary strategy for achieving this without human-provided labels.

• Goal: Transform high-dimensional, noisy data (like pixels or words) into a lower-dimensional latent space where semantic structure is preserved.
• Utility: Good representations are transferable; they can be used as input features for a variety of downstream tasks (classification, detection) with minimal fine-tuning.
• Example: A model pre-trained via self-supervision on millions of images learns a representation where "cat" and "dog" are closer in latent space than "cat" and "car," even without ever seeing those labels.

The lower-dimensional, continuous vector space where a model's learned representations reside. It is the output of an encoder in a representation learning system.

• Properties: A well-structured latent space captures the essential factors of variation in the data, allowing for meaningful operations.
• Key Operations:
  • Interpolation: Moving smoothly between two points (e.g., morphing a smiling face to a frowning face).
  • Arithmetic: Performing analogies (e.g., king - man + woman = queen in word embeddings).
• Disentanglement: An ideal latent space is disentangled, meaning each dimension corresponds to a single, interpretable factor (e.g., pose, lighting, identity).

An auxiliary, automatically generated task used to train a model in a self-supervised manner. The goal is not to excel at the pretext task itself, but to force the model to learn useful representations as a byproduct.

• Design Principle: The task must require understanding the data's inherent structure to solve.
• Common Examples:
  • Masked Language Modeling: Predicting a masked word from its context (BERT).
  • Jigsaw Puzzle: Reordering shuffled image patches.
  • Rotation Prediction: Predicting the angle by which an image was rotated.
  • Temporal Order Verification: Determining if two video frames are in the correct chronological order.

A reinforcement learning paradigm where an agent learns an explicit world model—a simulator of environment dynamics—and uses it for planning. This world model is often learned via self-supervision from the agent's experience.

• Core Loop: The agent interacts with the real environment, stores experiences, and uses them to self-supervise the training of its internal dynamics model (predicting next state and reward).
• Planning: The agent uses the learned model (e.g., via Model Predictive Control or Monte Carlo Tree Search) to simulate future trajectories and select optimal actions.
• Benefit: Dramatically improves sample efficiency compared to model-free RL, as the model allows for "thinking" without costly real-world interaction.