Weight Decomposition

Low-Rank Adaptation (LoRA) decomposes a weight update matrix ΔW ∈ ℝ^(d×k) into the product of two significantly smaller, low-rank matrices: ΔW = BA, where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with rank r ≪ min(d, k). This exploits the hypothesis that the update to pre-trained weights has a low intrinsic rank. During training, only A and B are updated, while the original weights W are frozen. The adapted forward pass becomes: h = Wx + BAx.

• Core Mechanism: Rank decomposition via matrix factorization.
• Key Advantage: Reduces trainable parameters from d×k to r×(d+k), often by >10,000x.
• Example: Adapting a 7B parameter LLM with LoRA rank r=8 may train only ~4M parameters.

Sparse decomposition methods, such as Diff Pruning, learn a binary or continuous mask that selects a sparse subset of the original parameters for updating. The weight update is decomposed as ΔW = M ⊙ Θ, where M is a sparse mask (often with <5% non-zero entries) and Θ is a dense task-specific diff. Only Θ and the mask parameters are trained.

• Core Mechanism: Element-wise masking to induce sparse fine-tuning.
• Key Advantage: Directly updates a small fraction of the original weights, preserving the base model's structure exactly.
• Example: For a weight matrix with 1M entries, a mask with 0.1% sparsity results in only 1,000 trainable diff parameters.

Additive scalar decomposition, exemplified by (IA)^3, learns task-specific scaling vectors that decompose the update into element-wise multiplications. It adapts activations or weight outputs by learning vectors l_k, l_v, l_ff that scale the keys, values, and feed-forward outputs in a transformer: AdaptedAttention(Q,K,V) = Attention(Q, l_k ⊙ K, l_v ⊙ V).

• Core Mechanism: Additive parameterization via learned scaling factors.
• Key Advantage: Extremely parameter-efficient; introduces only three vectors per transformer block.
• Example: For a block with hidden size d=768, (IA)^3 adds only 3*768=2,304 trainable parameters, acting as a form of conditional computation.

Bottleneck decomposition inserts small, dense feed-forward networks (adapters) into a model. An adapter decomposes the adaptation function into a down-projection matrix W_down ∈ ℝ^(d×r), a non-linearity, and an up-projection matrix W_up ∈ ℝ^(r×d), with bottleneck dimension r ≪ d. The adapter's output is added to the residual stream: h' = h + W_up(σ(W_down(h))).

• Core Mechanism: Modular adaptation via bottleneck neural modules.
• Key Advantage: Isolates task-specific knowledge into composable modules, enabling techniques like AdapterFusion and AdapterSoup.
• Example: A parallel adapter runs concurrently with the main layer, while a residual adapter is inserted sequentially.

Hypernetwork decomposition uses a secondary neural network (the hypernetwork) to generate the weights for the adapter or delta matrices of the main model. The decomposition is: ΔW = Hypernetwork(z), where z is a intrinsic task vector. This separates the task representation (z) from the parameter generation mechanism.

• Core Mechanism: Conditional computation via weight generation.
• Key Advantage: Enables efficient multi-task learning and rapid adaptation to new tasks by simply feeding a new task vector z.
• Example: A single hypernetwork can generate unique LoRA matrices (A, B) for hundreds of tasks, storing only the small task vectors z.

In Mixture-of-Experts (MoE) decomposition, a model's dense feed-forward layer is decomposed into multiple smaller expert layers. A gating network selects a sparse combination of experts for each input. The effective computation is a weighted sum: h' = Σ_i G(x)_i * E_i(x), where G(x) is the sparse gate and E_i are the experts.

• Core Mechanism: Sparse upcycling of dense layers into a sparse combination of specialized sub-networks.
• Key Advantage: Increases model capacity (e.g., trillion parameters) while keeping computational cost per token constant.
• Example: Sparse upcycling converts a dense pre-trained model into an MoE by splitting its FFN weights into multiple experts, reusing the original frozen backbone.

Weight decomposition enables a single base model to serve multiple downstream tasks efficiently. By learning separate, factorized delta weights (e.g., low-rank matrices for each task), the system can switch between specialized behaviors without storing multiple full-model copies.

• Key Benefit: Drastically reduces storage overhead. For a 70B parameter model, storing 10 full fine-tuned copies requires ~2.6TB. With LoRA-style decomposition (rank=8), storage drops to ~4.5GB.
• Use Case: A customer service platform uses one base LLM with decomposed task modules for intent classification, sentiment analysis, summarization, and policy retrieval, loading only the relevant deltas per query.

Decomposed weight updates are small enough to be downloaded and applied on resource-constrained devices, enabling federated fine-tuning and local personalization.

• Mechanism: A global model is deployed to devices. Local data generates a small low-rank update ΔW = BA. Only the factors (B, A) are sent to the cloud for secure aggregation, not raw data.
• Example: A smartphone keyboard model personalizes its autocorrect and next-word prediction by learning a user-specific low-rank decomposition on-device, preserving privacy and minimizing bandwidth.

Factorized deltas enable algebraic operations on model behavior, a principle known as task arithmetic. Decomposed task vectors can be added, subtracted, or interpolated.

• Process: If ΔW_sentiment and ΔW_formality are low-rank decompositions for those attributes, a new model for 'formal sentiment' can be created as W_new = W_base + αΔW_sentiment + βΔW_formality.
• Application: A content moderation system can quickly compose a filter by combining decomposed vectors for detecting toxicity, bias, and off-topic content, enabling rapid policy iteration without retraining.

By constraining updates to a low-dimensional subspace via decomposition, weight decomposition inherently limits interference with previously learned knowledge stored in the frozen backbone.

• How it Works: When learning Task B, the rank decomposition of its update ΔW_B exists in a different subspace from the decomposition for Task A. This structural sparsity minimizes overwriting.
• Real-World Impact: A medical imaging model can be sequentially adapted for detecting pneumonia, then tumors, then fractures, with each new capability stored as a compact module, preserving high accuracy on all tasks.

Searching over fine-tuning configurations (learning rate, rank, target modules) is exponentially cheaper when the trainable parameter count is reduced by 100-10,000x via decomposition.

• Efficiency Gain: A full fine-tuning hyperparameter sweep for a large model may be computationally prohibitive. A sweep over LoRA rank and alpha parameters trains only millions, not billions, of parameters per experiment.
• Enterprise Application: An AI platform can efficiently A/B test dozens of specialized adapter configurations for different client verticals (legal, finance, healthcare) using a shared base model, optimizing performance per domain.

Weight decomposition is the enabling technique behind modular adaptation paradigms like AdapterFusion and Mixture-of-Experts (MoE). It allows skills to be developed, tested, and composed independently.

• AdapterFusion: Independent low-rank adapters are trained for skills like translation, deduction, and coding. A fusion layer then learns to dynamically combine these expert layers.
• Sparse Upcycling: A dense pre-trained model can be 'upcycled' into an MoE system by decomposing its feed-forward weights into multiple expert candidates, reusing existing knowledge to bootstrap a more powerful, sparse architecture.

Low-Rank Adaptation (LoRA) is a foundational weight decomposition technique. It hypothesizes that weight updates during adaptation have a low intrinsic rank. Instead of updating the full weight matrix (W \in \mathbb{R}^{d \times k}), LoRA represents the update (\Delta W) as the product of two low-rank matrices: (\Delta W = BA), where (B \in \mathbb{R}^{d \times r}), (A \in \mathbb{R}^{r \times k}), and the rank (r \ll \min(d, k)). Only (A) and (B) are trained, leading to a massive reduction in trainable parameters.

• Core Mechanism: Applies low-rank updates to query, key, value, and output projection matrices in Transformer models.
• Key Benefit: The decomposed matrices can be merged with the base weights for zero-inference-overhead deployment.

Additive parameterization is the mathematical framework underpinning delta tuning methods like LoRA. It defines the adapted weights (W') as the sum of the original, frozen pre-trained weights (W_0) and a learned delta matrix (\Delta W). Formally: (W' = W_0 + \Delta W).

• Foundation for Decomposition: This formulation explicitly separates the static base knowledge ((W_0)) from the task-specific adaptation ((\Delta W)).
• Enables Efficiency: The delta (\Delta W) is the target for structured compression techniques, such as low-rank or sparse decomposition, making it parameter-efficient to learn and store.

Rank decomposition is the linear algebra operation of factorizing a matrix into a product of two or more matrices of lower rank. In the context of PEFT, it is applied to the weight update matrix (\Delta W).

• Mathematical Basis: For a matrix (\Delta W) of rank (r), it can be exactly decomposed as (\Delta W = U\Sigma V^T), where (U) and (V) are orthogonal matrices and (\Sigma) is a diagonal matrix of singular values.
• Approximation in LoRA: LoRA uses a truncated, low-rank approximation, keeping only the top (r) singular values and vectors, which captures the most significant directions of change for adaptation.

Sparse fine-tuning is a complementary strategy to low-rank decomposition for achieving parameter efficiency. Instead of a dense, low-rank update, it updates only a small, strategically selected subset of the model's parameters, leaving the vast majority at their pre-trained values.

• Methods: Includes techniques like Diff Pruning, which learns a sparse binary mask over the parameters, and BitFit, which updates only the bias terms.
• Contrast with Decomposition: While weight decomposition assumes updates are structured (low-rank), sparse fine-tuning assumes they are localized to a small fraction of parameters. The two principles can be combined.

A hypernetwork is a neural network that generates the weights for another neural network (the main model). In PEFT, a small hypernetwork can be trained to generate the decomposed weight updates (e.g., the (A) and (B) matrices for LoRA) conditioned on a task identifier or input.

• Dynamic Adaptation: Allows for conditional, input-specific adaptation without storing separate parameters for every task.
• Intrinsic Task Vectors: The hypernetwork is often conditioned on low-dimensional intrinsic task vectors, which are learned embeddings that compactly represent different tasks or skills.

A task vector is a mathematical construct derived from fine-tuning, defined as the difference between the fine-tuned and base model weights: (\tau = \theta_{ft} - \theta_{base}). This vector encapsulates the "direction" of adaptation for a specific task.

• Connection to Weight Decomposition: The task vector (\tau) is analogous to the learned delta (\Delta W). Efficient decomposition methods aim to represent this vector compactly.
• Task Arithmetic: Enables model editing by linearly combining task vectors (e.g., (\theta_{new} = \theta_{base} + \tau_{task1} + \tau_{task2})). This requires the delta representations to be stable and composable.