Sparse Importance Scoring

This is the simplest and most computationally efficient heuristic. It ranks parameters based on the absolute value of their pre-trained weights. The underlying assumption is that weights with larger magnitudes have a greater influence on the model's output and are therefore more important for task adaptation.

• Key Insight: Larger weights typically represent stronger learned connections from pre-training.
• Common Variants: L1 norm (absolute value), L2 norm (squared value).
• Limitation: Does not consider the new task's data; a large weight may be irrelevant for the target domain.

This method scores parameters by the magnitude of their gradients computed on the new task's data. A parameter with a large gradient indicates that changing it would cause a significant reduction in the task-specific loss, marking it as important for adaptation.

• Process: Perform a forward/backward pass on a batch of target task data and record gradient norms.
• Advantage: Dynamically reflects the parameter's relevance to the specific new task.
• Consideration: Gradients are noisy and can vary between batches; often requires aggregation over multiple steps.

A more sophisticated approach that uses the diagonal of the Hessian matrix (second derivatives of the loss with respect to parameters). The Hessian approximates the curvature of the loss landscape. Parameters with a high Hessian diagonal value are considered sensitive—moving them slightly causes a large change in loss—and are thus high-priority for tuning.

• Theoretical Basis: Related to optimal brain surgeon and Fisher information.
• Computational Cost: Calculating the full Hessian is prohibitive; efficient approximations (e.g., diagonal Fisher, AdaHessian) are used.
• Use Case: Favored for high-stakes selective pruning and fine-tuning where accuracy is paramount.

A statistically-grounded importance metric derived from the Fisher Information Matrix (FIM). For a parameter, its Fisher information is the expected square of its gradient, measuring how much the model's output distribution changes with a change in that parameter. High Fisher information implies high importance.

• Formula: (F_i = \mathbb{E}[ (\partial \log p(y|x) / \partial \theta_i)^2 ])
• Connection to Hessian: Under certain conditions, the FIM approximates the Hessian.
• Application: Foundational for Elastic Weight Consolidation (EWC) in continual learning and advanced sparse selection.

This heuristic scores parameters by observing how much they change during an initial phase of fine-tuning. Parameters that exhibit large movement (delta) from their pre-trained values are deemed important for the new task. This is a post-hoc measurement of importance.

• Methodology: Perform a few epochs of standard fine-tuning, then rank parameters by the absolute value of their weight change ((|\theta_{ft} - \theta_{pre}|)).
• Intuition: Weights that need to adjust significantly are crucial for task adaptation.
• Downside: Requires an initial training phase, adding overhead before sparse tuning can begin.

Instead of using a fixed heuristic, this approach parameterizes the selection process. A trainable mask (e.g., via a gating function with Gumbel-Softmax or LoRA-style gates) is applied to parameters. The mask values are optimized alongside the loss, allowing the model to learn which parameters are important for the task.

• Key Technique: Differentiable sparsity via relaxed binary masks.
• Advantage: Can discover complex, non-obvious importance patterns not captured by simple heuristics.
• Examples: Diff Pruning, Sparse Adapters with learned connectivity.
• Complexity: Introduces additional hyperparameters and training dynamics to manage.

A parameter-efficient adaptation technique that updates only a strategically selected, small subset of a pre-trained model's weights during training. This contrasts with full fine-tuning, which updates all parameters.

• Core Mechanism: Applies a binary or continuous mask to the model's gradient updates.
• Objective: Achieves task-specific performance while minimizing the number of altered parameters, reducing compute cost and memory footprint.
• Example: Fine-tuning a 175B parameter LLM by updating only 0.1% of its weights (175M parameters).

A parameter-efficient adaptation strategy that identifies and trains only the most important or task-relevant parameters within a pre-trained model. It is the broader paradigm that sparse importance scoring enables.

• Key Distinction: Focuses on the selection criteria (e.g., gradient magnitude, Hessian diagonal) to determine which parameters to update.
• Goal: Maximize adaptation efficiency by focusing compute on parameters with the highest expected impact on the loss function for the new task.

A core technique in sparse fine-tuning where a binary mask is applied to the model's weights or gradients to selectively freeze or enable updates to specific parameters.

• Implementation: The mask M is element-wise multiplied with the gradient ∇L during backpropagation: update = optimizer(∇L ⊙ M).
• Types: Can be static (determined once before training) or dynamic (potentially updated during training).
• Outcome: Creates a sparse update pattern, leaving the majority of the pre-trained model unchanged.

A class of gradient-based optimization algorithms, such as sparse SGD or sparse Adam, designed to efficiently handle models with a large proportion of zero-valued gradients or parameters.

• Engineering Challenge: Standard optimizers like Adam maintain momentum and variance estimates for all parameters, wasting memory on masked weights.
• Solution: Sparse optimizers only update and maintain state for the unmasked subset of parameters, leading to significant memory savings.
• Use Case: Essential for making sparse fine-tuning computationally practical for very large models.

A specific parameter-efficient fine-tuning method that learns a sparse, task-specific 'diff' vector representing the change from the pre-trained weights, with the diff regularized to be largely zero.

• Formulation: The fine-tuned weights are W_ft = W_pt + δ, where δ is the diff vector. A sparsity-inducing regularizer (e.g., L0 or L1) is applied to δ.
• Advantage: Explicitly encodes the adaptation as a sparse modification, making model merging and analysis more straightforward.
• Result: Produces a compact representation of the task adaptation that can be stored and applied separately from the base model.

The difference between a fine-tuned model's weights and its pre-trained base weights, where the vector is encouraged or constrained to be sparse. This enables efficient model merging and composition.

• Definition: For a base model θ_base and a fine-tuned model θ_ft, the task vector is τ = θ_ft - θ_base.
• Sparse Application: In sparse fine-tuning, τ is highly sparse, with non-zero values only for the selected parameters.
• Utility: Sparse task vectors can be arithmetically combined (e.g., added, subtracted) to create multi-task models or negate unwanted behaviors, a technique known as Task Arithmetic.