Sparse Diff Pruning

The core mechanism is learning a task-specific diff vector, Δθ, which is added to the frozen pre-trained weights, θ₀, to produce the adapted model: θ = θ₀ + Δθ. The key innovation is that Δθ is explicitly regularized during training to be sparse, meaning most of its elements are forced to zero. This results in a final model where only a small, critical subset of parameters deviates from their original pre-trained values, achieving high efficiency.

Inducing sparsity in the diff vector is typically achieved via L₀ regularization. Unlike L1 regularization which encourages small values, L₀ regularization directly penalizes the number of non-zero parameters. In practice, this is implemented using a hard concrete distribution or a straight-through estimator to make the binary mask decisions (zero vs. non-zero) differentiable. This allows the model to learn not just how much to change a parameter, but also whether to change it at all.

The method inherently performs automated parameter selection. Through the gradient-based optimization under L₀ regularization, the algorithm learns which weights are most important to update for the target task. This is a data-driven alternative to heuristic selection methods (e.g., magnitude pruning). The resulting sparse mask identifies a winning ticket-like subnetwork within the pre-trained model that is sufficient for adaptation.

A major advantage is efficient model storage and composition. Instead of saving a full set of fine-tuned weights (size of θ), one only needs to store the small base model θ₀ and the sparse diff vector Δθ (containing values and indices). This enables techniques like sparse task arithmetic, where diff vectors from multiple tasks can be added or averaged with minimal interference, facilitating multi-task learning and model merging.

Sparse diff pruning provides strong empirical evidence for a sparse variant of the Lottery Ticket Hypothesis. It demonstrates that for a given downstream task, there exists a sparse, trainable subnetwork (the 'diff') within the dense pre-trained model. When this subnetwork is activated and tuned, it can match or exceed the performance of fully fine-tuning the entire network, supporting the idea of intrinsic sparse adaptability in overparameterized models.

It differs fundamentally from adapter-based PEFT like LoRA or Adapters. Those methods add new parameters (small matrices or modules) to the model architecture. Sparse diff pruning modifies existing parameters but does so sparsely. It changes the base model's weights directly but selectively, offering a more native integration without additional inference latency from extra modules, though it requires more sophisticated optimization.

Sparse fine-tuning is the overarching paradigm of updating only a small, selected subset of a pre-trained model's weights. Unlike full fine-tuning, it strategically freezes the majority of parameters, drastically reducing memory footprint and compute cost. Methods include:

• Parameter Masking: Applying a binary mask to freeze/unfreeze weights.
• Gradient Masking: Zeroing out gradients for non-selected parameters during backpropagation.
• Selective Layer/Neuron Tuning: Updating only specific architectural components.

A sparse task vector is the mathematical difference (ΔW) between a fine-tuned model's weights and its original pre-trained weights, where ΔW is constrained to be largely zero. This is the core output of Sparse Diff Pruning.

Key Properties:

• Enables efficient model merging via arithmetic (e.g., adding task vectors).
• Facilitates multi-task composition by combining sparse vectors.
• Provides a compact, interpretable representation of task-specific knowledge.

Sparse importance scoring is the process of ranking model parameters to decide which weights are most critical for adaptation. This guides the creation of the sparse mask in diff pruning.

Common Heuristics:

• Magnitude-based: Weights with largest absolute change during early training steps.
• Gradient-based: Parameters with the highest gradient norms.
• Fisher Information: Estimates parameter importance via the expected gradient squared.
• Hessian-based: Uses second-order derivatives to measure loss sensitivity.

Sparse optimization refers to specialized variants of gradient-based algorithms (e.g., SGD, Adam) designed to handle models where most gradients are zero. These algorithms skip computations for masked parameters, leading to significant speedups.

Implementation Considerations:

• Requires efficient sparse tensor operations in frameworks like PyTorch.
• Often uses masked gradients where updates are only applied to an active subset.
• Critical for making sparse fine-tuning methods computationally viable.

Sparse model merging is the technique of combining multiple models, each adapted with sparse updates (like sparse diff pruning), into a single unified model. This leverages the sparsity of task vectors to avoid interference.

Advanced Methods:

• Task Arithmetic: Simple addition of sparse task vectors to the base model.
• TIES-Merging: Trims spurious changes, Elects sign consensus, and performs Sparse Disjoint Merging.
• Enables building multi-task models without additional training.

The Sparse Lottery Ticket Hypothesis posits that within a dense, pre-trained network, there exists a sparse subnetwork (a 'winning ticket') that, when trained in isolation, can match the full model's performance on a new task. This connects to sparse diff pruning's goal of finding a minimal, effective parameter subset.

Relation to PEFT:

• Sparse diff pruning can be seen as finding a winning ticket for the target task.
• Techniques like sparse supermasks (identifying masks without weight training) explore a related, extreme form of efficiency.