Model Merging (PEFT)

The foundational operation for model merging. A task vector is calculated as the arithmetic difference between a fine-tuned model's weights and the original pre-trained base model's weights (Δ = W_finetuned - W_base). Merging involves performing linear operations on these vectors.

• Averaging: Combining vectors from similar tasks (Δ_merged = (Δ_A + Δ_B) / 2) to improve robustness.
• Interpolation: Creating a weighted sum (Δ_merged = α * Δ_A + (1-α) * Δ_B) to balance task performance.
• Negation: Subtracting a vector (W_new = W_base - Δ) to potentially remove undesired behaviors or "unlearn" a task.

A state-of-the-art method that addresses interference from conflicting parameter signs across different task vectors. It performs three key steps:

• TrIm: Retains only the top-k% most significant parameters in each task vector, sparsifying the updates.
• Elect Sign: For each parameter, resolves sign conflicts by electing the majority sign across all vectors.
• Disjoint Merge: Averages only the parameter values that agree with the elected sign, reducing destructive interference.

This method enables the stable merging of a large number of diverse models, significantly outperforming simple averaging.

A technique designed to merge models fine-tuned with Low-Rank Adaptation (LoRA). It addresses the redundancy and overlap in LoRA delta weights.

• Random Drop: A large percentage (e.g., 90%) of delta weights are randomly set to zero.
• Rescaling: The remaining non-zero weights are rescaled (e.g., by 10x) to preserve the norm of the original delta.
• Merging: The sparsified and rescaled deltas are then averaged.

DARE allows for the lossless merging of dozens of LoRA-tuned models without performance degradation, as the dropped parameters are largely redundant.

An interpolation technique used when merging models, preferred over linear interpolation for certain parameter spaces. It interpolates along the geodesic (shortest path) on a hypersphere, treating weight sets as vectors.

• Use Case: Particularly effective for merging models whose fine-tuned weights have similar magnitudes but different directions in the high-dimensional parameter space.
• Process: Given two model weight vectors A and B, Slerp interpolates at angle θ, providing a smoother and more natural transition between model behaviors than linear interpolation (Lerp).
• Application: Commonly used in merging diffusion models or foundational LLMs to create balanced blends of capabilities.

Methods for creating a unified model from multiple fine-tuned checkpoints.

• Uniform Soup: The simplest form, averaging the weights of multiple models fine-tuned from the same base with different hyperparameters or data orders.
• Greedy Soup: Iteratively adds a model to the soup only if it improves validation performance on a target task.
• Gradient Souping: An advanced technique that merges models by approximating the task vectors that would result from fine-tuning on a mixture of all source tasks simultaneously. It computes a weighted average of gradients from each task to construct a more coherent merged model.

A data-driven merging approach that frames merging as a regression problem. Instead of purely geometric operations on weights, it uses a small calibration dataset to learn the optimal linear combination of multiple model outputs.

• Process: A lightweight regression layer (e.g., linear) is trained to combine the logits or hidden states of several frozen, task-specific models.
• Advantage: Directly optimizes for performance on the target mixture of skills, often yielding better results than weight-space arithmetic.
• PEFT Context: Highly compatible with merged PEFT modules, where the regression layer learns to weight the contributions of different adapters or LoRA modules.

Delta weights are the small set of learned parameter changes (ΔW) applied to a frozen pre-trained model during parameter-efficient fine-tuning. They represent the task-specific adaptation.

• In PEFT, only the delta weights are trained and stored, not the entire model.
• These are the fundamental unit for model merging, as merging operations are performed on the deltas from different tasks.
• Storing only deltas is highly storage-efficient, enabling the maintenance of many task-specific adaptations from a single base model.

A task vector is the arithmetic difference between the weights of a fine-tuned model and its original pre-trained base model. It mathematically encapsulates the knowledge acquired for a specific task.

• Calculated as: Task Vector = Fine-tuned Weights - Base Weights.
• Enables operations like model merging (adding task vectors) and task negation (subtracting task vectors).
• In PEFT, the task vector is often sparse or low-rank, corresponding directly to the trained adapter or LoRA matrices.

AdapterFusion is a two-stage PEFT method designed for multi-task learning. It first trains multiple independent, task-specific adapters, then learns a composition layer that dynamically combines them.

• Stage 1: Train lightweight adapters for N different tasks on a frozen backbone.
• Stage 2: Freeze the adapters and train a new fusion layer that learns to query and combine them for a new target task.
• This is a structured approach to model merging, allowing the composite model to leverage knowledge from multiple source tasks without catastrophic interference.

Model soup is a merging technique where the weights of multiple models fine-tuned from the same pre-trained checkpoint are averaged to create a single, more robust model.

• Performs a simple arithmetic mean of the weight parameters from different fine-tuned checkpoints.
• Often improves generalization and out-of-distribution robustness compared to any single constituent model.
• With PEFT, creating a soup is highly efficient because you average only the small delta weights or adapter parameters, not the entire massive backbone.

Task arithmetic is a framework for editing models by adding and subtracting task vectors. It treats model adaptation as operations in weight space.

• Addition (Base + Vector_A + Vector_B) merges capabilities from multiple tasks.
• Negation (Base + Vector_A - Vector_B) can remove an undesired skill or bias.
• Scaling (Base + λ * Vector_A) adjusts the strength of a task's influence.
• This provides a principled, mathematical basis for model merging and editing using the compact representations learned via PEFT methods.

Multi-Task PEFT refers to strategies for adapting a single pre-trained model to perform well on multiple downstream tasks simultaneously, using parameter-efficient methods.

• Approaches include training a shared set of adapters on a mixed multi-task dataset, or using merging techniques like AdapterFusion.
• Contrasts with sequential model merging, which combines models after single-task training.
• The goal is to achieve high performance across all tasks while adding only a minimal number of parameters per task, avoiding the need to store many separate models.