Projection Head

A projection head is typically a small multi-layer perceptron (MLP) with one to three hidden layers. In the standard SimCLR formulation, it consists of a two-layer MLP with a ReLU activation in between: Linear → ReLU → Linear. The output dimensionality is often smaller than the backbone's representation, commonly mapped to 128 or 256 dimensions. This bottleneck forces the encoder to retain only the most salient features. The head is non-linear to prevent the loss function from directly manipulating the representation space, which would degrade the quality of learned features.

The defining operational characteristic of a projection head is that it is discarded entirely after self-supervised pre-training. Only the backbone encoder's representations are used for downstream tasks like few-shot modulation recognition or emitter identification. Research in the SimCLR paper demonstrated that the representations before the projection head (the encoder output) are significantly more generalizable than those after it. The projection head absorbs invariance to data augmentations, which is useful for the contrastive objective but strips away information critical for discriminative downstream tasks.

Projection heads play a critical role in preventing dimensional collapse, where the encoder maps all inputs to a trivial constant or a low-dimensional subspace. By projecting to a higher or differently structured space before applying the loss, the head allows the encoder to maintain richer feature diversity. In methods like Barlow Twins and VICReg, the projector's output is where covariance and variance regularization terms are explicitly computed to enforce that different dimensions capture distinct information, ensuring the representation does not collapse to a non-informative solution.

The projection head maps encoder outputs to the space where contrastive loss functions operate. In InfoNCE loss, the normalized outputs of the projection head are used to compute cosine similarities between positive and negative pairs. The head's normalization layer (often L2 normalization) projects representations onto a unit hypersphere, where the temperature-scaled cross-entropy loss maximizes agreement between augmented views of the same sample while repelling views of different samples. This spherical embedding space is optimal for contrastive learning.

The depth of the projection head significantly impacts representation quality. The original SimCLR paper found that a 2-layer MLP outperformed both a linear projection and a deeper 3-layer head. A non-linear projection head improves representation quality by +10% to +20% on downstream tasks compared to a linear head or no head at all. However, overly deep heads can overfit to the pretext task. In BYOL, a deeper 2-layer projector is used in both the online and target networks, with the target network receiving a stop-gradient to prevent collapse.

In Radio Frequency Machine Learning, projection heads are adapted to handle complex-valued IQ data. The backbone encoder processes raw in-phase and quadrature (IQ) samples, and the projection head maps the resulting feature vector to a real-valued embedding space for contrastive loss. For few-shot modulation recognition, the projection head is pre-trained on massive unlabeled spectrum captures and then discarded, leaving a backbone that produces highly discriminative embeddings for prototypical networks to classify novel modulation schemes with as few as 5 labeled examples per class.

Modern self-supervised methods eliminate the need for negative pairs, but still rely on a projection head to prevent collapse.

• BYOL: Uses a momentum encoder and a predictor on top of the online network's projection head to learn without negatives.
• SimSiam: Demonstrates that stop-gradient is the critical mechanism, not momentum, for preventing collapse.
• Barlow Twins & VICReg: Replace contrastive loss with variance and covariance regularization applied to the projection head's output embeddings.

The projection head is typically a small multi-layer perceptron (MLP) with specific design choices that impact representation quality.

• Depth: Usually 2-3 layers, with a hidden dimension matching or exceeding the encoder's output.
• Bottleneck: A narrow middle layer forces the head to discard information irrelevant to the pretext task, preserving general features in the encoder.
• Output Dimension: Often 128 or 256 dimensions, regardless of the encoder's output size. This compact space is where the loss is computed.

The projection head is discarded after pre-training. The representations from the frozen encoder backbone are used directly for downstream tasks.

• Why it works: The head absorbs invariance to augmentations, allowing the encoder to retain richer, more generalizable features.
• Validation: Linear evaluation on the encoder's output consistently outperforms using the head's output for classification.
• RF Application: In few-shot modulation recognition, the encoder's output is fed to a prototypical network for classification with minimal labeled samples.

A constant risk in self-supervised learning is representation collapse, where the encoder outputs a trivial constant vector for all inputs. The projection head plays a key role in mitigation.

• Contrastive methods: The head's output space is structured by the loss to separate all samples, preventing collapse.
• Variance regularization: Explicitly penalizes low standard deviation in the head's output batch.
• Covariance regularization: Decorrelates feature dimensions to prevent informational collapse.
• Asymmetric architectures: Using a predictor and stop-gradient on one branch prevents the system from finding a collapsed shortcut solution.

The projection head is trained on a pretext task—a surrogate objective where labels are generated automatically from the data structure.

• Instance Discrimination: Treats each sample as its own class; augmented views must be identified among a batch of negatives.
• Masked IQ Modeling: Reconstructs intentionally blanked portions of an RF spectrogram, forcing the encoder to learn spectral structure.
• Temporal Ordering: Predicts the correct sequence of shuffled signal segments, learning temporal dynamics in raw IQ streams.
• Jigsaw Puzzles: Reorders shuffled patches of a spectrogram, teaching spatial relationships in time-frequency representations.