Fine-Tuning

Unlike training from scratch with random weights, fine-tuning begins with a pre-trained checkpoint. This provides a vastly superior starting point in the loss landscape. The model has already learned general, reusable features (e.g., edge detectors in vision, syntax in language), which accelerates convergence and requires significantly less target-domain data to reach optimal performance.

A core technique to prevent catastrophic forgetting of general knowledge. Early layers, which capture universal features, are often frozen (weights kept constant). Later, task-specific layers are trained with higher learning rates. Common strategies include:

• Discriminative fine-tuning: using a learning rate schedule that decays exponentially from the last to the first layer.
• Gradual unfreezing: progressively unfreezing layers from the top down during training.

The primary risk in fine-tuning is that the model overwrites its useful pre-trained knowledge with new, task-specific information. Mitigation strategies include elastic weight consolidation (EWC), which penalizes large changes to parameters deemed important for previous tasks, and experience replay, where a small subset of the original pre-training data is mixed into the target dataset to maintain the model's broad generalization capabilities.

The final classification or regression layer of the pre-trained model is invariably replaced with a new, randomly initialized head tailored to the target task. For example, a 1,000-class ImageNet classifier's head is swapped for a binary classifier for a defect detection task. The rest of the network acts as a frozen or slowly-trained feature extractor, transforming raw input into a rich feature vector that the new head learns to interpret.

Full fine-tuning updates every weight, creating a full model copy per task—prohibitively expensive for large models. PEFT methods like LoRA (Low-Rank Adaptation) inject small, trainable rank-decomposition matrices into frozen layers, often updating less than 1% of total parameters. This drastically reduces memory footprint and storage costs while matching full fine-tuning performance.

Fine-tuning is most valuable when the target dataset is small. However, a powerful pre-trained model can easily overfit to a tiny dataset, memorizing noise instead of learning the underlying pattern. Countermeasures include aggressive data augmentation (e.g., adding synthetic channel impairments for RF tasks), high dropout rates, and strong weight decay regularization to constrain the model's effective capacity.

The foundational paradigm where a model developed for one task is reused as the starting point for a second task. Fine-tuning is the specific execution step within transfer learning where the pre-trained weights are unfrozen and updated on a target dataset. This leverages generic features—like edge detection in vision or syntax in language—learned from massive source datasets, dramatically reducing the need for labeled target data.

A family of adaptation methods that update only a small fraction of model parameters, avoiding the compute and storage costs of full fine-tuning. Key techniques include:

• LoRA (Low-Rank Adaptation): Injects trainable rank decomposition matrices into attention layers.
• Adapters: Insert small bottleneck layers between existing layers.
• Prefix Tuning: Prepends learnable virtual tokens to the input sequence. PEFT is essential for adapting large models on consumer hardware.

The tendency of a neural network to abruptly and completely forget previously learned knowledge upon learning new information. During fine-tuning, if the target dataset is too narrow or the learning rate is too high, the model's weights can be overwritten, destroying the general representations learned during pre-training. Mitigation strategies include elastic weight consolidation, experience replay, and careful learning rate scheduling.

A subfield of transfer learning specifically addressing the distribution shift between a labeled source domain and an unlabeled or sparsely labeled target domain. Fine-tuning is a common technique for supervised domain adaptation, but adversarial methods like Gradient Reversal Layers can also be used to force the feature extractor to learn domain-invariant representations without explicit target labels.

Two distinct strategies for using a pre-trained model:

• Feature Extraction: The pre-trained backbone is frozen and treated as a fixed feature extractor. Only a new, randomly initialized classifier head is trained. This is computationally cheap but cannot adapt to significant domain gaps.
• Fine-Tuning: Selected layers of the pre-trained backbone are unfrozen and updated. This allows the model to shift its internal representations but carries a higher risk of overfitting on small datasets.

A critical hyperparameter strategy for fine-tuning. Common approaches include:

• Discriminative Fine-Tuning: Applying different learning rates to different layers, typically lower rates for early layers and higher rates for later, task-specific layers.
• Slanted Triangular Learning Rates: A schedule that quickly increases the learning rate and then slowly decays it, allowing the model to rapidly converge to a suitable region of the parameter space before fine-grained optimization.
• Cosine Annealing: Decays the learning rate following a cosine curve, often with warm restarts.