Delta Tuning

The core concept is representing adaptation as a parameter delta (ΔW), the mathematical difference between the fine-tuned weights and the original pre-trained weights. Instead of modifying W directly, delta tuning methods learn a compact, task-specific ΔW. The forward pass becomes: Output = f(x; W + ΔW). This separates the massive, static world knowledge (W) from the small, adaptable task knowledge (ΔW), enabling efficient storage and composition of multiple skills.

Directly learning a full ΔW for a model with billions of parameters is intractable. Delta tuning methods therefore impose a structural constraint or low-dimensional reparameterization on ΔW to make it learnable. Common decompositions include:

• Low-Rank Matrices: As in LoRA, where ΔW = BA, with B and A being low-rank.
• Sparse Masks: As in BitFit, where ΔW is non-zero only for bias terms.
• Scaling Vectors: As in IA³, where ΔW is a diagonal matrix (element-wise scaling). These constraints reduce trainable parameters by >90-99% while preserving adaptation capacity.

A key advantage of the delta formulation is the ability to compose and manipulate task vectors. Because deltas are additive, skills can be combined through arithmetic operations:

• Averaging: ΔW_multi = (ΔW_taskA + ΔW_taskB) / 2 creates a model with blended capabilities.
• Negation: W - ΔW_toxicity can be used to remove undesired behaviors.
• Scaling: α * ΔW can strengthen or weaken a particular adaptation. This enables model merging and the creation of multi-task models without retraining, a principle formalized as task arithmetic.

By keeping the base model frozen and only learning a small delta, these methods inherently protect the model's original generalization capabilities. This mitigates catastrophic forgetting, a common issue in full fine-tuning where task-specific learning overwrites fundamental linguistic or visual representations. The delta acts as a minimal, non-destructive override, allowing the model to retain performance on its original pre-training distribution while gaining new skills. This is critical for continual learning and building reliable multi-purpose systems.

Delta tuning provides a unifying framework to understand diverse PEFT techniques. Most methods are specific instantiations of learning a parameter delta under different constraints:

• Adapter Layers: ΔW is the function learned by a small inserted feed-forward network.
• Prefix/Prompt Tuning: ΔW is implicitly applied via modifications to the key-value cache or input embeddings.
• LoRA/DoRA: ΔW is explicitly parameterized as low-rank matrices.
• IA³: ΔW is a set of learned scaling vectors. This perspective allows researchers to compare methods by analyzing the expressivity and efficiency of their chosen delta parameterization.

The small size of learned deltas (often <1% of original model size) translates to direct engineering benefits:

• Storage: Thousands of task-specific adapters can be stored for the cost of one full model.
• Switching: Models can switch tasks in milliseconds by loading a different delta file.
• Distribution: Deltas can be shared and downloaded independently of the base model, which may be proprietary or too large to distribute.
• On-Device Personalization: Small deltas are feasible to train and deploy on edge devices (on-device adaptation), enabling local model personalization without sending data to the cloud.

Low-Rank Adaptation (LoRA) is a foundational delta tuning method. It hypothesizes that weight updates during adaptation have a low "intrinsic rank." Instead of updating a full weight matrix W (size d x k), LoRA injects two much smaller, trainable matrices A and B. The update is represented as W + ΔW = W + BA, where B is d x r, A is r x k, and the rank r is very small (e.g., 4, 8, 16). This reduces trainable parameters by thousands of times while often matching full fine-tuning performance.

A task vector is the literal mathematical representation of the "delta" in delta tuning. It is computed as the element-wise difference between the weights of a fine-tuned model and its original pre-trained base model: τ = θ_ft - θ_base. This vector encodes the directional change needed for a specific task. Key properties and uses include:

• Model Merging: Task vectors from different tasks can be added or interpolated (θ_merged = θ_base + ατ_A + βτ_B) to create multi-task models.
• Task Arithmetic: Scaling or negating task vectors can manipulate model behavior (e.g., adding a "politeness" vector, subtracting a "toxicity" vector).
• Efficiency: Storing only task vectors for many tasks is far more efficient than storing entire fine-tuned models.

An adapter is a small, self-contained neural network module inserted into the layers of a frozen pre-trained model. It is a classic implementation of delta tuning, where the "delta" is the entire adapter module's function. A standard design is a bottleneck:

• Down-Projection: A linear layer that projects the layer's hidden state to a lower dimension.
• Non-Linearity: An activation function like ReLU or GELU.
• Up-Projection: A linear layer that projects back to the original hidden dimension.
• Residual Connection: The adapter's output is added to the original hidden state. Only the adapter's parameters are trained, making it highly parameter-efficient. Adapters demonstrate the modular aspect of delta tuning, where task-specific knowledge is encapsulated in plug-and-play components.

Prefix tuning and prompt tuning are delta tuning methods that apply the parameter change to the model's input space rather than its internal weights. They learn a set of continuous, task-specific embeddings.

• Prefix Tuning: Prepends trainable vectors (the prefix) to the key and value states of every transformer layer's attention mechanism. This allows deep, layer-wise steering of the model's attention patterns.
• Prompt Tuning: Prepends trainable vectors only to the input embeddings at the first layer (the "soft prompt"). It is simpler and uses even fewer parameters. In both cases, the "delta" is this learned context that reprogram's the frozen model's behavior, showing that effective adaptation can be achieved without modifying any core weights.

Sparse fine-tuning is a delta tuning strategy where the "delta" is applied to only a strategically selected, sparse subset of the model's original parameters. Instead of adding new parameters (like LoRA or Adapters), a small fraction of the base model's existing weights are updated. Common approaches include:

• BitFit: Updates only the bias terms in the model, which typically constitute <0.1% of total parameters.
• Layer-wise Adaptation: Updates parameters only in specific, critical layers (e.g., the last few transformer blocks).
• Attention Head Masking: Learns which attention heads are most important for a task and selectively updates them. This approach is based on the insight that not all parameters are equally important for task adaptation, and significant gains can be achieved by updating a minimal, targeted set.

Model merging is a post-training technique that directly manipulates the deltas (task vectors) from multiple fine-tuned models. It operationalizes the additive property of task vectors to combine competencies without additional training. Core methods include:

• Task Arithmetic: Performs linear operations on task vectors: θ_merged = θ_base + Σ(α_i * τ_i). Scaling factors α_i can be tuned for performance.
• TIES-Merging: A robust method that addresses interference between merged task vectors by Trimming insignificant changes, Electing the sign of merged values, and performing Sparse disjoint merging.
• Adapter Composition: Merges models by combining their respective adapter modules, often using a learned gating mechanism (e.g., Mixture-of-Adaptors). Merging enables the creation of multi-task, generalist models from a collection of efficient, task-specific deltas.