Additive Parameterization

Additive parameterization defines the adapted weight matrix W' for a layer as W' = W₀ + ΔW, where W₀ represents the frozen, pre-trained weights and ΔW (delta W) is the learned task-specific update. This formulation explicitly separates the massive, static knowledge of the foundation model from the small, adaptable component, enabling efficient fine-tuning by optimizing only ΔW.

• Core Equation: The forward pass for a linear layer becomes y = (W₀ + ΔW)x + b.
• Frozen Base: W₀ remains unchanged, preserving the model's original capabilities and preventing catastrophic forgetting.
• Learned Delta: ΔW is the only component updated during training, drastically reducing the number of trainable parameters.

The primary advantage of additive parameterization is its extreme parameter efficiency. By design, the number of trainable parameters is decoupled from the size of the base model W₀. Techniques like Low-Rank Adaptation (LoRA) exploit this by imposing a structural constraint on ΔW, factorizing it as ΔW = BA, where B and A are low-rank matrices.

• Example: For a weight matrix W₀ of size d x k, a LoRA rank r (where r << min(d, k)) creates trainable matrices B (d x r) and A (r x k).
• Result: Trainable parameters drop from d * k to r * (d + k). For a large model, this can reduce trainable parameters by 10,000x or more.
• Memory Footprint: During training, only ΔW (or its factors) and optimizer states need to be stored in GPU memory for the adapted parameters, enabling fine-tuning of massive models on consumer hardware.

The additive nature of the parameterization enables powerful model editing and composition operations. Since adaptation is represented as a discrete delta, these deltas can be manipulated algebraically.

• Task Vectors: A fine-tuned delta ΔW_task can be treated as a task vector. Multiple task vectors can be combined via linear arithmetic: W' = W₀ + αΔW_A + βΔW_B.
• Applications: This allows for task addition (combining skills), task negation (removing undesired behaviors), and interpolation (blending model behaviors).
• AdapterSoup: A practical application where the parameters (ΔW) from multiple task-specific adapters are averaged for multi-task inference: W'_soup = W₀ + (1/N) Σ ΔW_i.

To maintain efficiency, the learned delta ΔW is not a full, dense matrix. Different PEFT methods apply distinct structural constraints to ΔW:

• Low-Rank (LoRA): ΔW = BA, a product of two low-rank matrices.
• Sparse (Diff Pruning): ΔW = m ⊙ Θ, where m is a sparse binary mask and Θ are small trainable parameters.
• Scalar (IA)^3: ΔW is represented as element-wise scaling vectors applied to activations or specific weight columns.
• Bias-Only (BitFit): ΔW is constrained to only the bias terms in the model.

These constraints are the parameterization in additive parameterization, determining how the delta is represented and learned while ensuring the update remains small and efficient.

A key operational benefit of additive parameterization is the ability to merge the delta back into the base weights for zero-latency inference. Since W' = W₀ + ΔW, the adapted model can be converted into a standard, monolithic model post-training.

• Process: The learned ΔW is added to W₀ in memory, creating a new W'. The adapter modules can then be discarded.
• Advantage: Eliminates the computational overhead of separate adapter forward passes, making inference latency and memory usage identical to the base model.
• Dynamic vs. Static: Methods like AdapterDrop use dynamic, input-dependent routing, but pure additive methods typically allow for static weight merging, simplifying deployment.

Additive parameterization is the underlying mechanism that enables modular adaptation. Each learned delta ΔW can be viewed as a self-contained module that modifies the base model's behavior.

• Module = Delta: A task-specific adapter implements a particular ΔW. The base model W₀ is the shared, frozen platform.
• Composition: Techniques like AdapterFusion learn to compose multiple pre-trained deltas (ΔW_1, ΔW_2, ...) for a new task.
• Hypernetworks: A hypernetwork can be seen as a meta-model that generates the delta ΔW = f_φ(z_task), where z_task is a task embedding, further abstracting the parameterization.

This perspective frames additive parameterization as the core enabler of a library of skills (deltas) that can be applied to a universal base model.

Delta weights (ΔW) are the learned parameter updates that constitute the additive change applied to a frozen pre-trained model. In additive parameterization, the adapted weights are computed as W' = W + ΔW, where W is the original frozen weight matrix. These deltas are the sole trainable parameters during fine-tuning.

• Core Principle: Represent the minimal, task-specific adjustment needed for adaptation.
• Efficiency: Their size is typically a small fraction of the base model's parameters.
• Implementation: In methods like LoRA, delta weights are factorized into low-rank matrices (ΔW = BA).

Low-Rank Adaptation (LoRA) is a seminal PEFT method that instantiates the additive parameterization principle. It hypothesizes that weight updates during adaptation have a low intrinsic rank. LoRA represents the delta matrix ΔW as the product of two low-rank matrices, ΔW = BA, where A and B are trainable.

• Key Innovation: Dramatically reduces the number of trainable parameters versus full fine-tuning.
• Additive Form: The adapted forward pass becomes h = Wx + BAx, clearly separating the frozen base computation from the learned delta.
• Widespread Use: The de facto standard for efficiently fine-tuning large language models.

A task vector is a mathematical construct that encapsulates the delta learned for a specific task. It is often derived by subtracting the pre-trained model's weights from the fine-tuned model's weights: τ = θ_ft - θ_pre. This vector directly embodies the additive change.

• Representation: Encodes the direction and magnitude of adaptation in weight space.
• Task Arithmetic: Task vectors can be combined through linear operations (e.g., addition, negation) to create models for new, composite tasks without additional training.
• Abstraction: Provides a high-level view of model editing and multi-task adaptation.

The frozen backbone is the large, pre-trained base model (e.g., GPT-4, LLaMA) whose parameters are kept entirely static during parameter-efficient fine-tuning. In additive parameterization, this backbone's weights (W) are the immutable foundation to which the delta (ΔW) is added.

• Stability: Preserves the model's general knowledge and prevents catastrophic forgetting.
• Efficiency: Eliminates the need to store optimizer states for the vast majority of parameters, drastically reducing memory overhead.
• Modularity: Enables multiple specialized adaptations (via different deltas) to share the same foundational model.

Weight decomposition is a general strategy for representing a complex weight update in a factorized, efficient form. Additive parameterization often relies on decomposition to make the delta ΔW parameter-efficient.

• Low-Rank Decomposition: Used by LoRA (ΔW = BA).
• Sparse Decomposition: Used by methods like Diff Pruning, where ΔW is represented as a sparse mask applied to a dense delta.
• Scalar Decomposition: Used by (IA)^3, where ΔW is decomposed into element-wise scaling vectors applied to activations.
• Goal: To model the essential adaptation signal with minimal parameters.

A residual adapter is a classic modular component that operationalizes additive parameterization within a neural network layer. It is a small feed-forward network inserted into a model that processes the hidden states and adds its output back to the residual stream.

• Architecture: Typically a down-projection, non-linearity, and up-projection (bottleneck).
• Additive Integration: The adapter's output is added to the layer's output: h' = h + f_adapter(h). This is a direct, localized form of additive parameterization.
• Contrast with Parallel Adapter: A residual adapter is usually inserted sequentially after a layer, whereas a parallel adapter runs concurrently with the layer's main feed-forward network.