Linear Probing

The core principle of linear probing is that the pre-trained backbone remains completely frozen during evaluation. No gradients flow back into the encoder; only the weights of a single linear layer are updated. This strict isolation ensures the evaluation measures the intrinsic separability of the learned representations rather than the model's capacity to adapt. In medical imaging, this is critical for determining if features learned on natural images (ImageNet) or via self-supervised pre-training on radiology data are immediately useful for a downstream task like pathology classification.

Linear probing accuracy serves as a canonical proxy metric for representation quality. A high linear probe score indicates that the pre-trained features form well-separated clusters in the embedding space, making them linearly separable. This metric is widely used in self-supervised learning benchmarks to compare methods like SimCLR, MoCo, and DINO without the confounding variable of fine-tuning hyperparameters. For medical imaging, a strong linear probe result on a target domain suggests the features have captured clinically relevant anatomical or pathological concepts.

Because only a single linear layer is trained, linear probing is orders of magnitude faster than full fine-tuning. The frozen backbone processes each image once to extract a fixed feature vector, which can be cached to disk. Training then reduces to solving a convex optimization problem on a relatively small feature matrix. This efficiency enables rapid experimentation across multiple pre-training checkpoints and downstream tasks. For 3D medical volumes where full fine-tuning is prohibitively expensive, linear probing provides a practical first-pass evaluation.

Linear probing has a well-documented limitation: it can underestimate the true potential of a pre-trained model. Representations that are not linearly separable may still contain rich, hierarchically organized features that a non-linear classifier or full fine-tuning could exploit. This discrepancy is known as the fine-tuning gap. In medical imaging, features encoding subtle textural patterns of early-stage lesions may require non-linear transformations to become diagnostically useful, making linear probing a conservative lower bound on transfer performance.

A rigorous linear probing evaluation follows a standardized protocol to ensure reproducibility:

• Feature extraction: Pass all target dataset images through the frozen encoder once.
• Normalization: Apply L2 normalization to the extracted feature vectors.
• Classifier training: Train a linear classifier (e.g., logistic regression or a single fully-connected layer) using cross-validated hyperparameters for learning rate and weight decay.
• No data augmentation: Unlike fine-tuning, augmentations are typically avoided to isolate representation quality from augmentation-induced invariance. This protocol is the de facto standard in self-supervised learning literature.

Linear probing is a powerful diagnostic tool for detecting negative transfer early. If a pre-trained model achieves lower linear probe accuracy on a target medical imaging task than a randomly initialized network trained end-to-end, it signals a fundamental domain mismatch. This can occur when pre-training data (e.g., natural photographs) shares no low-level statistical structure with the target domain (e.g., X-ray imagery). Identifying this mismatch via linear probing saves the computational cost and time of attempting futile fine-tuning runs.

Linear probing strictly decouples feature extraction from classification. The pre-trained encoder's weights are completely frozen, preventing any gradient flow backward. A single linear layer is then trained on the extracted embeddings. This isolates the evaluation to the quality of the representations themselves, removing confounding variables introduced by end-to-end fine-tuning.

The canonical workflow follows a strict sequence:

• Feature Extraction: Pass all target medical images through the frozen pre-trained encoder once to generate a static embedding dataset.
• Classifier Training: Train a linear classifier (e.g., logistic regression or a single fully-connected layer) on these frozen embeddings.
• No End-to-End Backprop: Crucially, gradients are never propagated back into the encoder. This ensures the test measures the pre-trained model's immediate utility.

Linear probing serves as a diagnostic tool to answer: 'How linearly separable are the learned features?' High probing accuracy indicates the pre-trained model has already learned a feature space where classes are well-clustered. This is particularly valuable in medical imaging for comparing self-supervised pre-training strategies on unlabeled DICOM data before committing to expensive fine-tuning experiments.

These two evaluation paradigms test fundamentally different properties:

• Linear Probing: Tests the immediate linear separability of frozen features. A strict measure of the pre-trained representation's quality.
• Fine-Tuning: Tests the adaptability of the pre-trained weights. The model can re-organize its feature space entirely. A large gap between fine-tuning and linear probing accuracy often indicates a significant domain shift between the source and target data.

Linear probing is the gold-standard evaluation metric in self-supervised learning (SSL) research. Protocols like MoCo, SimCLR, and DINO all report linear probing accuracy on downstream tasks. In medical imaging, this allows researchers to benchmark SSL pre-training methods on large, unlabeled PACS datasets by probing on smaller, annotated classification tasks like pathology detection.

Despite its prevalence, linear probing has known drawbacks:

• Underestimates Capacity: A frozen encoder may contain high-quality features that are not linearly separable but are easily extracted by a non-linear head.
• Misleading Comparisons: Methods that produce highly compressed, linearly separable features may score well on probing but perform worse after fine-tuning.
• Alternative: k-Nearest Neighbor (k-NN) evaluation on frozen features is a non-parametric alternative that avoids training a classifier entirely.

The process of unfreezing and updating all or part of a pre-trained network's weights on a target dataset. Unlike linear probing, fine-tuning allows the feature extractor itself to adapt to domain-specific patterns.

• Full fine-tuning: Updates every parameter, risking catastrophic forgetting
• Partial fine-tuning: Freezes early layers, updates later layers
• Requires significantly more compute and data than linear probing
• Often follows linear probing as a second-stage adaptation

A family of techniques designed to mitigate domain shift between source and target distributions. While linear probing evaluates feature transferability, domain adaptation actively transforms representations to close the gap.

• Supervised DA: Requires labeled target data
• Unsupervised DA: Uses only unlabeled target samples
• Domain-adversarial training uses a gradient reversal layer to learn domain-invariant features
• Critical when source models are trained on natural images but deployed on medical scans

A paradigm where models learn representations from unlabeled data by solving pretext tasks before any linear probing evaluation. This approach has revolutionized medical imaging where labels are scarce.

• Contrastive learning (SimCLR, MoCo): Pulls augmented views of the same image together
• Masked image modeling (MAE): Reconstructs hidden image patches
• Representations from self-supervised pre-training often outperform supervised ImageNet features on medical tasks
• Linear probing is the standard benchmark for evaluating these learned representations

Methods that adapt pre-trained models by training only a small fraction of parameters, bridging the gap between frozen linear probing and full fine-tuning. These techniques preserve the original feature space while enabling task-specific adaptation.

• LoRA (Low-Rank Adaptation): Injects trainable rank-decomposition matrices into attention layers
• Adapter modules: Lightweight bottleneck layers inserted between frozen layers
• Prompt tuning: Learns soft prompts in the input space
• Achieves near full fine-tuning performance with <1% trainable parameters

The tendency of neural networks to abruptly overwrite previously learned knowledge when adapted to a new task. Linear probing avoids this entirely by freezing the backbone, making it a safe evaluation protocol.

• Occurs because gradient updates for the new task interfere with weights encoding old knowledge
• Elastic Weight Consolidation (EWC) penalizes changes to important parameters
• Experience replay interleaves old task data during training
• In medical imaging, forgetting natural image features may be desirable if they interfere with diagnostic patterns

A training strategy that applies different learning rates to different layer groups during fine-tuning. This concept extends the linear probing philosophy by allowing controlled, graduated unfreezing of layers.

• Early layers (general features): Low learning rate (1e-5)
• Middle layers (transitional features): Medium learning rate (1e-4)
• Later layers (task-specific features): Higher learning rate (1e-3)
• Often combined with gradual unfreezing, where layers are unfrozen one group at a time
• Prevents destructive updates to general-purpose features learned during pre-training