Joint Embedding Architecture

The foundational architectural pattern where two or more identical encoder networks process different augmented views of the same input image. These parallel branches share weights, ensuring consistent representation extraction. In medical imaging, this typically employs a Vision Transformer (ViT) or ResNet backbone pre-initialized with domain-agnostic weights. The siamese design enforces that semantically identical anatomical structures—regardless of augmentation—map to proximate points in the embedding space.

A small multi-layer perceptron (MLP) attached to the encoder output that maps representations to a lower-dimensional space where the contrastive or redundancy-reduction loss is applied. Key characteristics:

• Typically 2-3 layers with ReLU activation and a linear output
• Hidden dimension often set to 2048 or 4096, with output dimension of 128-256
• Discarded after pre-training; only the backbone encoder is retained for downstream tasks
• Prevents the loss function from distorting the representation space used for transfer learning

A slowly evolving copy of the online encoder, updated via exponential moving average (EMA) rather than backpropagation. The update rule follows: θ_target ← m·θ_target + (1-m)·θ_online, where m is typically 0.99-0.999. This mechanism:

• Provides stable regression targets that do not fluctuate with each training step
• Decouples the target representation from the current mini-batch statistics
• Is critical in BYOL and MoCo architectures to prevent representation collapse
• Creates a form of temporal ensembling that smooths the learning signal

A critical architectural component in non-contrastive methods that blocks gradient flow through one branch of the joint embedding architecture. Implemented as a PyTorch-style detach() or TensorFlow stop_gradient() operation on the target network output. This asymmetric design:

• Breaks symmetry between the online and target branches
• Prevents both networks from collapsing to identical trivial solutions
• Forces the online network to predict the target's representations without the target adapting in response
• Is essential for BYOL, SimSiam, and DINO to function without negative pairs

A large memory bank of encoded representations maintained as a first-in-first-out (FIFO) queue, decoupling dictionary size from mini-batch size. Key properties:

• Stores the K most recent encoded keys (typically K=65,536 or larger)
• Enables access to a vast, consistent set of negative samples without massive batches
• Updated by enqueuing the current mini-batch and dequeuing the oldest entries
• The momentum encoder ensures stored keys remain consistent over time
• Core innovation of MoCo that makes contrastive learning computationally feasible on standard GPU hardware

An additional MLP attached only to the online branch in non-contrastive frameworks like BYOL and SimSiam. This predictor transforms the online network's output before comparing it to the target network's representation. The asymmetry introduced by the prediction head:

• Creates an information bottleneck that prevents trivial copying
• Forces the online network to learn a predictive mapping rather than an identity function
• Is typically a 2-layer MLP with batch normalization and a bottleneck hidden dimension
• Works in conjunction with stop-gradient to eliminate the need for negative pairs entirely

The foundational architectural pattern where two or more identical encoders with shared weights process different augmented views of the same input. This weight-sharing constraint is critical: it forces the model to produce consistent embeddings regardless of the applied transformation. In medical imaging, branches might process different intensity windows of a CT scan or multi-planar reconstructions of the same anatomy. The architecture terminates in a projection head that maps encoder outputs to the space where the invariance loss is computed.

The mathematical objective that drives representation learning by maximizing agreement between embeddings of transformed views. Key variants include:

• Cosine similarity loss: Minimizes the angle between positive pair embeddings
• Mean Squared Error: Used in BYOL-style methods to regress one view's representation to another's
• Cross-entropy on cluster assignments: SwAV's swapped prediction mechanism The loss must be carefully balanced with variance and covariance regularization terms to prevent the encoder from finding a trivial constant-output solution.

Non-contrastive methods introduce deliberate asymmetry to prevent representation collapse without relying on negative pairs. Key mechanisms include:

• Stop-gradient operation: Prevents backpropagation through the target branch, as in BYOL and SimSiam
• Momentum encoder: A slowly updating exponential moving average of the online network that provides stable target representations
• Predictor MLP: An additional network head on the online branch that transforms the representation before comparing it to the frozen target output These components break the symmetry that would otherwise allow both branches to output identical trivial vectors.

Explicit regularization techniques that prevent dimensional collapse, where all embeddings converge to a low-dimensional subspace. VICReg formalizes this with three loss terms:

• Variance loss: A hinge loss that maintains the standard deviation of each embedding dimension above a threshold, ensuring the representation spreads across the full vector space
• Covariance loss: Minimizes off-diagonal entries of the embedding covariance matrix, decorrelating features to maximize information content
• Invariance loss: Standard MSE between positive pairs Barlow Twins achieves similar decorrelation by pushing the cross-correlation matrix toward the identity matrix.

A data augmentation protocol that generates multiple views at different resolutions from a single medical image. Standard practice uses 2 large crops (e.g., 224×224 covering >50% of the image) and V small crops (e.g., 96×96 covering <30%). The loss enforces consistency between:

• Global-to-global: Large crop pairs capture overall anatomical context
• Global-to-local: Large crops matched with small crops force the model to relate fine-grained texture to whole-organ structure This is particularly effective for histopathology where cellular details must be understood within tissue architecture.

Methods like SwAV and DINO replace explicit pairwise comparison with cluster assignment consistency. The process:

• Compute soft cluster assignments (codes) for each view using a prototype matrix
• Enforce that different views of the same image map to similar assignment distributions
• Apply the Sinkhorn-Knopp algorithm to ensure balanced cluster usage across the batch, preventing the trivial solution where all samples collapse to a single cluster This implicitly compares each sample against all prototypes, providing an efficient alternative to large-scale negative sampling.