Variance Regularization

In self-supervised learning, the most efficient way to minimize a naive invariance loss is representation collapse—where the encoder outputs a constant vector for all inputs. This trivial solution carries zero information. Variance regularization explicitly forbids this by enforcing a minimum standard deviation on the embeddings within a batch, ensuring diverse outputs for diverse inputs.

The variance term is typically implemented as a hinge loss on the standard deviation:

• Computes the standard deviation of each embedding dimension across the batch
• Applies a penalty only if the standard deviation falls below a fixed threshold (e.g., 1.0)
• L_var = mean(ReLU(threshold - std)) This allows the model to freely maintain variance above the threshold without infinite pressure to increase it.

Variance regularization is a core component of the VICReg (Variance-Invariance-Covariance Regularization) framework, which combines three complementary loss terms:

• Variance: Prevents collapse by enforcing non-zero standard deviation
• Invariance: Minimizes distance between augmented views of the same sample
• Covariance: Decorrelates embedding dimensions to prevent informational redundancy Together, these terms produce rich, non-collapsed representations without requiring negative pairs.

Unlike contrastive methods that operate on pairs, variance regularization works across the entire batch dimension. For each dimension j of the embedding vector z, the standard deviation is computed over all N samples in the batch. This makes the technique sensitive to batch size—too small a batch can produce noisy variance estimates, while larger batches provide more stable regularization signals.

In self-supervised RF representation learning, variance regularization prevents the encoder from collapsing to a DC constant when processing raw IQ samples:

• Ensures the model distinguishes between different modulation types, SNR conditions, and emitter signatures
• Prevents the trivial solution of outputting the mean IQ value
• Works synergistically with augmentations like noise addition and frequency shifting
• Critical for training on unlabeled spectrum captures where collapse is a constant risk

Different self-supervised frameworks prevent collapse through distinct mechanisms:

• Contrastive (SimCLR): Uses explicit negative pairs to push apart dissimilar samples
• Momentum Encoder (BYOL): Relies on stop-gradient and EMA to avoid collapse
• Clustering (DeepCluster): Uses discrete pseudo-labels to maintain diversity
• Variance Regularization (VICReg): Directly constrains the embedding distribution Variance regularization is unique in providing an explicit, measurable guarantee against collapse.

Variance-Invariance-Covariance Regularization explicitly combines three loss terms to prevent collapse. The variance term is a hinge loss on the standard deviation of embeddings, the invariance term minimizes distance between augmented views, and the covariance term decorrelates feature dimensions. This joint optimization provides a robust, principled framework for non-contrastive self-supervised learning on RF data.

A complementary collapse-prevention technique that operates on the covariance matrix of embeddings within a batch. By penalizing the off-diagonal entries, it forces the encoder to produce features that are linearly decorrelated, preventing informational redundancy where multiple neurons encode the same signal characteristic. Often paired with variance regularization in methods like Barlow Twins and VICReg.

A self-supervised objective that makes the cross-correlation matrix between twin network embeddings as close to the identity matrix as possible. This simultaneously achieves:

• Invariance: diagonal entries forced to 1.0
• Redundancy reduction: off-diagonal entries forced to 0.0 Unlike explicit variance regularization, Barlow Twins prevents collapse by ensuring each feature component captures unique, non-redundant information about the RF signal.

A slowly evolving copy of the main encoder, updated via exponential moving average (EMA) rather than backpropagation. Used in frameworks like MoCo and BYOL, the momentum encoder produces stable target representations that prevent the online network from finding a trivial collapsed solution. The stop-gradient operation on the teacher branch is critical—without it, the system collapses to a constant output.

A paradigm where a student model is trained to predict the output of a teacher model sharing the same architecture. Central to non-contrastive methods like BYOL and SimSiam, self-distillation prevents collapse through architectural asymmetry—the teacher is updated via EMA while the student learns via gradient descent. The student must learn meaningful representations to match the teacher's evolving targets without trivial solutions.

The primary failure mode that variance regularization addresses. Collapse occurs when an encoder produces a constant, non-informative output for all inputs, achieving zero loss on invariance but learning nothing useful. Types include:

• Full collapse: all outputs identical
• Dimensional collapse: outputs span a low-dimensional subspace
• Informational collapse: features lack discriminative power Prevention strategies include variance regularization, covariance decorrelation, and contrastive negative sampling.