Sparse Task Vectors

A sparse task vector is formally defined as the element-wise difference between the weights of a model fine-tuned on a specific task and its original pre-trained weights: τ = θ_fine-tuned - θ_pre-trained. The core innovation is the application of a sparsity-inducing constraint or regularization (e.g., L1 norm) during fine-tuning, forcing the majority of elements in τ to be zero or near-zero. This results in a compact representation of the task-specific adaptation, where only a critical subset of parameters is altered.

The primary utility of sparsity is enabling model merging through simple arithmetic operations. Because task vectors are sparse and largely non-overlapping, multiple vectors can be added to a base model with minimal interference. For example, a model proficient in both translation and summarization can be created via θ_multi-task ≈ θ_base + τ_translation + τ_summarization. Sparsity is crucial here, as dense updates would cause catastrophic parameter interference, degrading performance on all tasks.

Sparse task vectors support task arithmetic, where vectors are combined to create models with composed capabilities. Key operations include:

• Addition: Combining skills (e.g., τ_french + τ_chemistry for French chemistry QA).
• Negation: Removing unwanted behaviors or biases.
• Interpolation: Scaling a vector's influence (e.g., θ_base + α * τ_task). The success of these operations relies on the sparse disjointness of vectors, meaning they modify largely distinct parameter sets, preventing destructive conflicts.

Achieving an effective sparse task vector requires specialized training techniques:

• Regularization: Applying L1 or L0 regularization to the weight delta during fine-tuning.
• Diff Pruning: Learning a sparse, task-specific 'diff' vector with a penalty on its L0 norm.
• Magnitude-Based Pruning: After standard fine-tuning, applying a threshold to zero out small changes in the task vector.
• Learned Masks: Training a binary or continuous mask that selects which parameters can be updated, often via a gating function with a sparsity penalty.

Sparsity translates directly to dramatic storage and memory savings. Instead of storing multiple multi-billion parameter models (e.g., one per task), an enterprise can store a single base model and many small, sparse task vectors. For a 7B parameter model with 0.1% sparsity, a task vector requires storing only ~7M significant values and their indices. This enables cost-effective deployment of numerous specialized model variants from a shared foundation.

Sparse task vectors are the output of sparse fine-tuning processes. Techniques like sparse diff pruning, sparse LoRA, and selective fine-tuning (e.g., updating only attention or MLP layers) are all methods to produce a sparse task vector. The vector's quality—its sparsity pattern and the utility of the updated weights—is determined by the selection heuristic (e.g., magnitude, gradient, Fisher information) used during the fine-tuning process.

Sparse fine-tuning is the overarching parameter-efficient adaptation technique where only a small, strategically selected subset of a pre-trained model's weights are updated during training. Unlike full fine-tuning, it avoids modifying the entire parameter set, which drastically reduces computational cost and memory footprint.

• Core Mechanism: Applies a binary or continuous mask to the model's gradients or weights, allowing updates to flow only to the unmasked parameters.
• Key Benefit: Preserves the bulk of the model's pre-trained knowledge while efficiently specializing it for a new task.
• Example: Updating only 5% of a 7-billion parameter model's weights for a specific customer service classification task.

Selective fine-tuning is a strategy that identifies and trains only the most important or task-relevant parameters within a pre-trained model. It is often synonymous with sparse fine-tuning but emphasizes the criteria used for selection.

• Selection Heuristics: Parameters are chosen based on metrics like weight magnitude, gradient norms, or approximations of the Fisher Information Matrix.
• Granularity: Can be applied at different levels: sparse neuron tuning (individual units), sparse layer tuning (entire layers), or sparse structured tuning (blocks of weights).
• Objective: Maximize task performance per parameter updated, ensuring compute is spent on the most impactful weights.

Parameter masking is the core technical implementation for sparse and selective fine-tuning. It involves applying a mask—typically a binary matrix—to the model's weights or gradients to selectively freeze or enable updates.

• Static vs. Learned Masks: A static mask is defined before training (e.g., via importance scoring). A sparse learned mask is a parameterized, trainable mask whose values are optimized during fine-tuning.
• Function: For a weight matrix W and mask M, the effective update is often computed as W = W_base + M ⊙ ΔW, where ⊙ is element-wise multiplication and ΔW is the weight delta.
• Outcome: Creates the sparse task vector, where the mask ensures the delta (ΔW) is non-zero only for the selected parameters.

Sparse diff pruning is a specific PEFT method that directly learns a sparse, task-specific 'diff' vector representing the change from the pre-trained weights. The diff is regularized to be largely zero, enforcing sparsity.

• Direct Delta Learning: The optimization objective minimizes task loss plus an L0 or L1 regularization penalty on the diff vector, pushing most delta values to zero.
• Connection to Sparse Task Vectors: This method explicitly produces a sparse task vector as its primary output.
• Advantage: Provides a clear, compressed representation of the adaptation, which is ideal for sparse model merging and storage.

Sparse model merging is the process of combining multiple models, each fine-tuned on different tasks with sparse parameter updates, into a unified multi-task model. It leverages the sparsity of task vectors to avoid interference.

• Process: Involves averaging the sparse task vectors (weight deltas) from multiple fine-tuned models. Because each vector is sparse and may affect different parameters, they can often be combined additively.
• Advanced Technique: Sparse TIES-Merging refines this by Trimming spurious changes, Electing a sign consensus, and performing disjoint merging.
• Use Case: Creating a single model capable of handling customer support, code generation, and document analysis by merging three sparse adapters.

Sparse importance scoring is the methodology for ranking a model's parameters to determine which weights are most critical for adaptation to a new task. It provides the foundation for selective fine-tuning.

• Common Metrics:
  • Sparse Magnitude Pruning: Uses the absolute value of pre-trained weights as a proxy for importance.
  • Sparse Hessian-based Selection: Uses the diagonal of the Hessian matrix to estimate a parameter's sensitivity to the loss.
  • Sparse Fisher Information: Measures how much information a parameter carries about the task distribution.
• Application: The top-k most important parameters according to the chosen score are selected for tuning, while the rest are frozen.