Fine-Tuning

The core mechanism of fine-tuning is transferring learned weights from a source model pre-trained on large-scale natural images (e.g., ImageNet) or medical datasets. Instead of starting from random weights, the model begins with hierarchical feature detectors already in place. Early layers capture universal features like edges, corners, and color blobs, while deeper layers encode more abstract, task-specific representations. This initialization provides a dramatically better starting point than random weights, enabling convergence with far fewer labeled medical images. The pre-trained weights act as a strong inductive bias, constraining the model to a region of the loss landscape that generalizes well.

Not all layers are updated equally during fine-tuning. Common strategies include:

• Full Fine-Tuning: All weights are unfrozen and updated on the target dataset. Highest capacity but risks catastrophic forgetting and overfitting on small datasets.
• Partial Fine-Tuning: Early layers are frozen to preserve generic feature extractors, while only later, task-specific layers are updated. This is the most common approach in medical imaging.
• Progressive Unfreezing: Layers are unfrozen one at a time from the top down, training each newly unfrozen layer before proceeding. This provides fine-grained control over the adaptation process.
• Discriminative Learning Rates: Different layer groups receive different learning rates, with earlier layers getting smaller updates to preserve pre-trained knowledge.

Catastrophic forgetting occurs when a fine-tuned model abruptly overwrites its previously learned representations, losing the generalization capabilities that made pre-training valuable. This is especially dangerous in medical imaging, where the target dataset is often small and highly specialized. If the learning rate is too high or too many layers are unfrozen, the model can overfit to the target domain's narrow distribution, forgetting robust features learned from diverse source data. Mitigation strategies include elastic weight consolidation, which penalizes large changes to important pre-trained weights, and experience replay, which interleaves source-domain examples during fine-tuning.

Fine-tuning demands careful learning rate control. Best practices include:

• Warmup Phase: A short period of linearly increasing learning rate to stabilize early updates and prevent destructive gradient steps.
• Cosine Annealing: A schedule that smoothly decays the learning rate following a cosine curve, allowing the model to settle into a sharp minimum.
• Discriminative Rates: Applying a lower learning rate (e.g., 1e-5) to frozen or early layers and a higher rate (e.g., 1e-4) to newly initialized task-specific heads.
• One-Cycle Policy: A single cycle of increasing then decreasing learning rate, which can accelerate convergence and act as implicit regularization. The goal is to update task-specific parameters aggressively while gently nudging generic feature extractors.

The final classification or regression head of the pre-trained model is almost always discarded and replaced with a new, randomly initialized head designed for the target medical task. For example, an ImageNet model with a 1000-class softmax head would have that head removed and replaced with a binary classifier for malignant vs. benign tumor detection. The new head is trained with a higher learning rate than the backbone, as it must learn from scratch. Common head architectures include global average pooling followed by a single dense layer for classification, or a U-Net decoder for segmentation tasks where the pre-trained encoder is used as the contracting path.

Medical imaging datasets are often orders of magnitude smaller than natural image datasets, making fine-tuning prone to overfitting. A model with millions of parameters can easily memorize a few hundred chest X-rays. Countermeasures include:

• Aggressive Data Augmentation: Applying random rotations, flips, intensity shifts, and elastic deformations to artificially expand the training set.
• Strong Weight Decay: L2 regularization to penalize large weight magnitudes.
• Dropout: Randomly dropping neurons during training to prevent co-adaptation.
• Early Stopping: Halting training when validation performance plateaus.
• Stochastic Weight Averaging (SWA): Averaging weights from multiple checkpoints to find a flatter, more generalizable minimum.

A fine-tuning strategy that applies different learning rates to distinct layer groups within the network. Early layers, which capture universal features like edges and corners, receive a lower learning rate to preserve general knowledge. Later, task-specific layers receive a higher rate to adapt quickly to the target domain.

• Typical ratio: 1x for final classifier head, 0.1x for middle blocks, 0.01x for initial convolutional layers
• Rationale: prevents the destruction of transferable features while allowing task-specific adaptation
• Implementation: commonly used with the slice method in fastai or custom optimizer param groups in PyTorch

An evaluation protocol where only a linear classifier is trained on top of frozen, pre-trained features. All backbone weights remain completely unchanged. This serves as a diagnostic tool to assess the quality of the learned representations without the confounding effects of full fine-tuning.

• Purpose: measures how linearly separable the pre-trained features are for the target task
• Contrast with fine-tuning: linear probing evaluates representation quality; fine-tuning maximizes task performance
• Medical use case: quickly benchmarking multiple pre-trained models on a new pathology classification dataset before committing to expensive full fine-tuning

A strategy for artificially expanding a limited training dataset by applying label-preserving transformations to combat overfitting during fine-tuning. Medical imaging requires domain-specific augmentations beyond standard flips and rotations.

• Intensity transforms: random contrast, brightness, and gamma adjustments to simulate scanner variability
• Spatial transforms: elastic deformations and affine transformations to model anatomical variation
• Advanced techniques: MixUp, CutMix, and CutOut adapted for 3D volumetric data
• Critical constraint: augmentations must preserve diagnostic content—a flipped chest X-ray with dextrocardia could create a false positive