Feature Decorrelation

The primary failure mode addressed by decorrelation is dimensional collapse, where the embedding space effectively shrinks to a lower-dimensional subspace. This happens when neurons become dead or highly correlated.

• Spectral Analysis: Collapse is diagnosed by a rapid drop in the singular values of the embedding matrix.
• Information Bottleneck: Collapse represents a degenerate solution where the encoder ignores input variance to minimize the loss.
• Decorrelation methods act as an information maximization regularizer, preserving the full representational capacity of the high-dimensional output space.
• This is critical for medical imaging, where subtle pathological features could be lost if the embedding space collapses.

Feature decorrelation methods offer a distinct paradigm from contrastive learning by operating on the feature dimension rather than the batch dimension.

• No Negative Pairs: They do not require explicit repulsion between dissimilar samples, simplifying the training pipeline.
• Batch Size Independence: Performance does not degrade with small batch sizes, unlike SimCLR which requires large batches for effective negative sampling.
• No Momentum Encoder: Unlike MoCo or BYOL, they typically use symmetric siamese networks without a slowly updating target branch.
• Computational Overhead: The covariance matrix calculation scales quadratically with the feature dimension d, which can be a bottleneck for very high-dimensional embeddings.

Decorrelation is highly effective for self-supervised pre-training on medical scans, where anatomical structures exhibit high spatial correlation that can lead to feature redundancy.

• 3D CT/MRI Volumes: VICReg and Barlow Twins can be applied to volumetric patches, ensuring the model learns distinct features for different tissue types and textures.
• Pathology: Whole slide images contain repetitive cellular patterns; decorrelation forces the model to learn a diverse set of morphological feature detectors.
• Cross-Modal Alignment: The redundancy-reduction principle can align features from different modalities (e.g., T1 and T2 MRI) by decorrelating shared and private information.
• These methods excel in label-scarce scenarios, learning robust representations from unlabeled DICOM archives before fine-tuning on small annotated datasets.

A critical failure mode in self-supervised learning where the encoder produces a constant, non-informative output for all inputs. Types include:

• Complete collapse: All outputs map to a single vector, achieving zero variance.
• Dimensional collapse: The embedding space collapses to a lower-dimensional subspace, where many dimensions carry zero information. Feature decorrelation directly combats dimensional collapse by ensuring each vector component captures independent information. Methods like BYOL and SimSiam prevent collapse through architectural stop-gradient operations rather than explicit covariance penalties.

A statistical transformation that converts a set of features into a representation where the covariance matrix equals the identity matrix and the mean is zero. This is the explicit, deterministic equivalent of what feature decorrelation objectives approximate through gradient-based optimization.

• ZCA Whitening: Minimizes the distortion to the original data while decorrelating features.
• Batch Whitening: Applied within neural network layers to decorrelate activations during training. Unlike Barlow Twins or VICReg, which softly penalize correlations, whitening enforces exact decorrelation through matrix decomposition, but introduces computational overhead from eigenvalue calculations.

A square matrix where element (i,j) represents the covariance between feature dimensions i and j across a batch of embeddings. The diagonal contains variances of individual features; off-diagonal elements capture redundancy between pairs of features. In feature decorrelation objectives:

• The goal is to drive off-diagonal elements toward zero.
• This ensures each dimension captures unique, non-redundant information.
• The matrix is typically computed on centered embeddings (zero-mean) for numerical stability. The size of this matrix scales as d × d, where d is the embedding dimension, making it a computational bottleneck for high-dimensional representations.

The Noise-Contrastive Estimation objective that maximizes mutual information between positive pairs by framing representation learning as a categorical classification problem. The model must identify the true positive sample among a set of K negative distractors. While InfoNCE does not explicitly decorrelate features, it implicitly encourages diverse representations by requiring the model to distinguish between many negative examples. However, without sufficient negative diversity, InfoNCE-trained models can still suffer from dimensional collapse, which is why methods like Barlow Twins and VICReg were developed as explicit redundancy-reduction alternatives.