Frozen Backbone

The core principle of a frozen backbone is that the original parameters of the pre-trained model remain entirely unchanged during fine-tuning. This preserves the general knowledge—such as linguistic syntax, visual features, or cross-modal alignments—acquired during large-scale pre-training on diverse datasets. The model's foundational representations are treated as a stable, reusable asset.

By freezing the backbone, training updates are restricted to a tiny fraction of the total model parameters. For example:

• Adapters may train only 0.5-8% of the original parameter count.
• LoRA often trains less than 1% of the weights. This drastically reduces VRAM consumption, storage overhead for checkpoints, and training time, making adaptation of billion-parameter models feasible on consumer-grade hardware.

Freezing the backbone acts as a strong regularizer, preventing catastrophic forgetting—the phenomenon where a model loses previously learned general capabilities when trained on a new, narrow task. The frozen weights ensure the model's original performance on broad benchmarks is retained, while the new, efficient modules learn the task-specific adaptation.

Efficient modules are inserted at specific injection points within the frozen architecture. Common locations include:

• After the attention mechanism in a transformer block.
• After the feed-forward network.
• Within projection layers for cross-modal models. These modules, such as adapters or LoRA matrices, learn to transform the backbone's intermediate activations for the new task without altering the core computational path.

A single frozen backbone can support multiple, independent efficient modules for different tasks. This enables:

• Multi-task serving from one base model.
• Continual learning by adding new adapters for new tasks without interfering with old ones.
• Model composition through techniques like AdapterFusion, which blends knowledge from multiple task-specific adapters.

The frozen backbone paradigm is modality-agnostic. Standard implementations include:

• Encoder Models: Frozen BERT backbones with BERT Adapters for NLP.
• Vision Models: Frozen Vision Transformers (ViTs) with ViT Adapters for segmentation.
• Multimodal Models: Frozen CLIP or BLIP backbones with VL-Adapters for vision-language tasks.
• Audio Models: Frozen Wav2Vec2 models with Audio Adapters for speech tasks.

The small set of learned parameter changes (Δ) applied to a frozen pre-trained model during PEFT. These weights represent the task-specific adaptation.

• Core Concept: The mathematical difference between the final adapted model and the original backbone.
• Storage Efficiency: Only the delta weights need to be saved and loaded, drastically reducing storage overhead compared to full model checkpoints.
• Example: In LoRA, the delta weights are the low-rank matrices A and B that approximate the update ΔW = BA.

The specific architectural locations within a neural network where parameter-efficient modules are inserted to interface with the frozen backbone.

• Common Locations: After the multi-head attention module or the feed-forward network in a transformer layer.
• Design Choice: The choice of injection point (e.g., parallel vs. sequential adapter placement) affects gradient flow and task performance.
• Multimodal Context: In vision-language models, adapters may be injected into the vision encoder, text encoder, or cross-attention fusion layers.

The tiny subset of a model's total parameters that are updated during PEFT, while the backbone remains frozen.

• Efficiency Metric: Typically <1-10% of the original model's parameter count.
• Includes: Adapter weights, prompt embeddings, low-rank matrices, bias terms (in BitFit), or scaling vectors (in IA³).
• Impact: Enables rapid fine-tuning, reduces memory footprint, and mitigates catastrophic forgetting by preserving the backbone's pre-trained knowledge.

The arithmetic difference between the weights of a fine-tuned model and its pre-trained base model, encapsulating the knowledge acquired for a specific task.

• Representation: Task Vector = θ_fine-tuned - θ_pre-trained.
• Applications: Enables model merging via vector arithmetic (e.g., adding task vectors) and task negation (e.g., subtracting an unwanted behavior vector).
• Research Frontier: Used in techniques like model soup and task arithmetic to build multi-task models from a collection of PEFT checkpoints.

A key hyperparameter in adapter-based PEFT that controls the size of the adapter's hidden layer, determining its capacity and parameter count.

• Function: Creates a computational bottleneck: projects the input dimension down, applies a non-linearity, then projects back up.
• Trade-off: A smaller bottleneck increases parameter efficiency but may reduce adaptation capacity. A typical reduction factor is 16 or 32.
• Formula: For an input dimension d, the adapter adds roughly 2 * d * (d / r) parameters, where r is the reduction factor.

The process of combining the delta weights or task vectors from multiple models, each fine-tuned on a different task, into a single cohesive model.

• Mechanism: Performs element-wise arithmetic (e.g., averaging, addition) on the learned adapters or task vectors.
• Benefit: Achieves multi-task capabilities or improved generalization without expensive multi-task training.
• Challenge: Requires careful weighting and can lead to interference if tasks are not compatible. Methods like TIES-Merging help resolve conflicts.