Representation Collapse

A direct constraint that penalizes the encoder when the standard deviation of embeddings within a batch falls below a threshold. VICReg implements this as an explicit hinge-loss term on the variance of each dimension, forcing the network to produce diverse outputs across different input samples. Without this, the encoder can trivially collapse by mapping all IQ samples to the same vector. In RF applications, this ensures the model distinguishes between noise floors, modulation types, and emitter signatures rather than ignoring input variations.

Prevents informational redundancy by decorrelating the features of learned embeddings. The objective minimizes the off-diagonal entries of the covariance matrix computed over a batch, pushing different dimensions of the representation vector to capture independent signal characteristics. Barlow Twins achieves this by making the cross-correlation matrix between twin network embeddings close to the identity matrix. In RFML, this forces the encoder to disentangle attributes like carrier frequency offset, symbol rate, and pulse shaping rather than encoding the same information redundantly across dimensions.

A critical architectural component in self-distillation frameworks like BYOL and SimSiam that blocks gradient flow to the teacher or target network. The student network learns to predict the teacher's output, but the teacher is updated only via exponential moving average (EMA) of the student's weights—never by backpropagation. This asymmetry prevents the model from finding the trivial collapsed solution where both networks output identical constant vectors. The stop-gradient breaks the symmetry that would otherwise allow mutual collapse.

A slowly evolving copy of the main encoder, updated via exponential moving average: θ_teacher ← τθ_teacher + (1−τ)θ_student, where τ is typically 0.999–0.9999. Used in MoCo and BYOL, the momentum encoder produces stable, consistent target representations across training steps. This temporal ensembling smooths out noisy updates and prevents the target from drifting toward collapse. In RF self-supervised pre-training, the momentum encoder provides high-quality targets for contrastive or predictive objectives applied to sequential IQ streams.

Frameworks like SimCLR and MoCo explicitly prevent collapse by pushing apart representations of dissimilar samples (negative pairs) while pulling together augmented views of the same sample (positive pairs). The InfoNCE loss mathematically requires a sufficiently large and diverse set of negatives to avoid the encoder mapping all inputs to a single point. MoCo maintains a dynamic queue of thousands of negative embeddings, decoupled from the mini-batch size, enabling robust contrastive learning on RF datasets where batch sizes may be limited by GPU memory constraints.

Introducing architectural differences between the online and target branches prevents collapse by ensuring the two networks cannot trivially agree on a constant output. BYOL uses a predictor MLP on top of the online network that the target network lacks, creating an asymmetric mapping. The online network must predict the target's representation through this additional projection, making the constant-output solution non-optimal. In RF applications, this asymmetry encourages the encoder to capture meaningful signal structure rather than exploiting shortcut symmetries.