Self-Supervised Pre-training

The pretext task is a self-supervised objective where the model learns by solving a puzzle derived from the data itself, not from human labels. In RF domains, this often involves predicting the future of a signal (Contrastive Predictive Coding), reconstructing masked IQ samples (Masked Autoencoder), or identifying augmented versions of the same signal (SimCLR). The task must be sufficiently difficult to force the model to learn high-level semantic features of the signal structure rather than trivial shortcuts.

Self-supervised methods split into two paradigms:

• Contrastive methods (SimCLR, MoCo, CPC) pull positive pairs together and push negative pairs apart in embedding space, requiring careful negative sampling strategies like the momentum encoder queue.
• Non-contrastive methods (BYOL, Barlow Twins, VICReg) eliminate negative pairs entirely, instead preventing representation collapse through architectural tricks like stop-gradient operations and statistical regularization on the embedding covariance matrix.

A critical failure mode where the encoder outputs a constant vector for all inputs, achieving zero loss on the pretext task but learning nothing useful. Prevention strategies include:

• Variance regularization: Penalizing low standard deviation across the batch dimension.
• Covariance regularization: Decorrelating feature dimensions to prevent informational redundancy.
• Momentum encoders: Using a slowly evolving EMA of the online network to provide stable, non-collapsing targets in frameworks like BYOL and MoCo.

A small MLP attached to the backbone encoder during pre-training that maps representations to a space where the contrastive or self-distillation loss is applied. Crucially, the projection head is discarded after pre-training. The representation used for downstream tasks (modulation recognition, emitter identification) is taken from the layer immediately preceding the projection head, as it retains more general signal features that transfer better to new tasks.

In the absence of labels, data augmentations define the invariances the model must learn. For RF signals, augmentations include adding synthetic noise, applying small frequency offsets, simulating channel fading, or using MixUp IQ and CutMix IQ to blend samples. The model learns that these perturbed versions should map to similar representations, forcing it to ignore nuisance channel variations and focus on the intrinsic signal structure.

The true measure of pre-training quality is performance on a downstream task with limited labels. Standard evaluation protocols include:

• Linear probing: Freezing the backbone and training only a linear classifier on top.
• Few-shot fine-tuning: Adapting the entire model with only 1-10 labeled examples per class using Prototypical Networks or standard SGD. Strong performance in the low-label regime indicates the pre-trained representations have captured meaningful signal semantics.

A failure mode in self-supervised learning where the encoder produces a constant or non-informative output for all inputs. Collapse renders the pre-trained model useless for downstream tasks and is the central challenge SSL methods must overcome.

• Dimensional collapse: all representations lie in a low-dimensional subspace
• Complete collapse: encoder outputs identical vector regardless of input
• Prevented by: contrastive negative sampling, variance regularization, covariance regularization, or stop-gradient mechanisms
• RF-specific risk: strong narrowband interference can dominate representations and induce collapse