Trainable Components

This is the foundational mathematical principle for delta tuning. Adapted weights (W') are represented as the sum of the original frozen weights (W₀) and a learned delta matrix (ΔW): W' = W₀ + ΔW. The trainable component is the delta ΔW, which is typically constrained to be low-rank or sparse to minimize parameters. This formulation ensures the pre-trained knowledge in W₀ is preserved as a stable foundation.

Trainable components are designed to operate in a significantly reduced parameter space compared to the full model. For example:

• LoRA represents ΔW as the product of two low-rank matrices (A and B), where the rank r is often less than 8.
• Adapters use a bottleneck architecture, projecting activations down to a hidden dimension (e.g., 64) before projecting back up. This constraint is the primary source of efficiency, reducing memory footprint and speeding up training.

Many PEFT methods enforce sparsity in which parameters are updated. The trainable component is not a dense update to all weights, but a targeted change:

• Sparse Fine-Tuning / Diff Pruning: Learns a binary mask selecting a small percentage (<0.1%) of the original model's weights to update.
• BitFit: Updates only the bias terms within the transformer.
• Selective Fine-Tuning: Uses heuristics or saliency scores to identify and update only the most task-relevant layers or parameters.

Trainable components like adapters are designed as self-contained, plug-and-play modules. This enables:

• Multi-Task Learning: Training separate, small adapters for different tasks on the same frozen backbone.
• Adapter Composition: Techniques like AdapterFusion learn to combine multiple pre-trained adapters for a new task.
• AdapterSoup: Averaging the parameters of multiple task-specific adapters at inference for robust multi-task performance without extra training.

In advanced architectures, trainable components are activated based on the input or task, a form of conditional computation. This maximizes efficiency by only using necessary compute for a given input.

• Mixture-of-Experts (MoE): A gating network routes each token to a sparse combination of expert layers (which can be considered large trainable components).
• Hypernetworks: A small auxiliary network generates the weights for the trainable component (e.g., an adapter) conditioned on a task vector, allowing dynamic specialization.

A core design goal is to minimize interference with the base model's general knowledge. By freezing the backbone and constraining updates to small, additive, or modular components, catastrophic forgetting is largely avoided. The trainable component learns a task-specific deviation from the base model's behavior rather than overwriting its fundamental representations. This makes PEFT models more stable and suitable for continual learning scenarios.

Adapter modules are small, bottleneck feed-forward neural networks inserted sequentially or in parallel within transformer blocks. They are the quintessential trainable component in modular adaptation.

• Structure: Typically consist of a down-projection, a non-linearity (e.g., ReLU), and an up-projection.
• Placement: Commonly inserted after the feed-forward network or the multi-head attention module within a transformer layer.
• Parameter Efficiency: A single adapter may add only 0.5-8% of the base model's parameters per task. For a 7B parameter model, this equates to 35-560 million trainable parameters, a massive reduction from full fine-tuning.

In Low-Rank Adaptation (LoRA), the trainable components are pairs of low-rank matrices that approximate the full weight update. For a weight matrix W (of dimension d x k), LoRA represents the update as ΔW = B * A, where B is d x r, A is r x k, and the rank r is significantly smaller than d and k.

• Key Insight: The intrinsic rank of weight updates during adaptation is often very low (e.g., rank r=4, 8, or 16).
• Trainable Parameters: Only the matrices A and B are updated. For query/value projections in attention layers of a large model, this can reduce trainable parameters by >10,000x compared to full fine-tuning.
• Deployment: The low-rank matrices can be merged with the frozen weights for zero-inference-overhead deployment.

Prompt tuning and prefix tuning optimize continuous, task-specific embeddings prepended to the model's input or hidden states. These vectors are the sole trainable components.

• Soft Prompts: A small set of trainable token embeddings (e.g., 20-100 virtual tokens) prepended to the input sequence. The model's embedding table and all other weights remain frozen.
• Prefix Vectors: Trainable tensors prepended to the key and value caches of the transformer's attention mechanism at every layer, steering the model's contextual processing.
• Efficiency: For a model with a 4096-dimensional hidden size and 30 layers, a prefix of length 10 adds only ~1.2 million trainable parameters, a minuscule fraction of a multi-billion parameter model.

The (IA)^3 method introduces trainable task-specific scaling vectors that modulate inner activations. These vectors are the only components updated during training.

• Mechanism: Learns three sets of vectors: one to scale the keys (K), one for the values (V) in attention, and one for the intermediate feed-forward network (FFN) activations.
• Parameter Count: Adds only three vectors per transformer layer, each with a dimension equal to the model's hidden size. For a 24-layer model with a 1024-dimensional hidden state, this is just 73,728 trainable parameters.
• Operation: The scaling is performed via element-wise multiplication, making it computationally lightweight during both training and inference.

In sparse fine-tuning methods like Diff Pruning, the trainable component is a sparse binary mask (and sometimes accompanying small diff weights) applied to a subset of the pre-trained model's parameters.

• Function: The mask M selects which original parameters θ receive a small, learned delta δ. The adapted weight is θ' = θ + M ⊙ δ.
• Sparsity: The mask is highly sparse, often targeting <0.5% of the model's total parameters for update. In a 1B parameter model, this means <5 million parameters are trainable.
• Learning: The mask itself can be learned via a differentiable relaxation (e.g., the Hard Concrete distribution) or pre-determined by a saliency criterion.

BitFit is an extreme PEFT method where the only trainable components are the bias terms within the transformer architecture. All weight matrices (e.g., in linear projections, attention, feed-forward networks) remain completely frozen.

• Scope: Updates only the bias parameters in layers like attention query/key/value projections, feed-forward networks, and layer norms.
• Extreme Efficiency: For many transformer models, biases constitute only ~0.1% of the total parameters. Fine-tuning a 110M parameter BERT model with BitFit updates only ~140,000 parameters.
• Performance: Surprisingly effective for many natural language understanding tasks, demonstrating that re-calibrating activation offsets via biases can be a powerful adaptation signal.

Delta weights are the learned parameter updates, or the difference (ΔW), applied to the frozen weights of a pre-trained model during delta tuning to achieve task-specific adaptation. They represent the core trainable component in methods like LoRA, where the full weight update is constrained to a low-rank factorization.

• Core Concept: The additive change ΔW such that the adapted weights are W_adapted = W_base + ΔW.
• Efficiency: The parameters of ΔW are the only ones updated, constituting a tiny fraction of the full model.
• Example: In LoRA, ΔW is decomposed into two low-rank matrices A and B, where ΔW = BA.

Additive parameterization is a method of representing a model's adapted weights as the sum of the original frozen weights and a learned delta matrix. This is the foundational mathematical principle behind delta tuning techniques, explicitly separating the stable base knowledge from the new, adaptable component.

• Formula: W' = W₀ + Δ(W).
• Advantage: Prevents catastrophic forgetting by preserving W₀ and isolates new learning to Δ(W).
• Implementation: Used directly in LoRA, (IA)³, and is the conceptual basis for adapter modules that add residuals to activations.

A residual adapter is a small module inserted into a neural network that operates on the residual stream, typically applying a bottleneck feed-forward network whose output is added back to the hidden states. It is a canonical example of a trainable component in modular adaptation.

• Architecture: Consists of a down-projection, a non-linearity, and an up-projection, creating a bottleneck (e.g., reducing dimension to 64).
• Insertion Point: Placed sequentially after the feed-forward network or attention module in a transformer block.
• Mechanism: Computes an adaptation Δh that is added to the layer's output: h' = h + FeedForward_Bottleneck(h).

A parallel adapter is an adapter module whose computation runs in parallel with the original layer's feed-forward network, with its output added to the main residual stream. This contrasts with sequential adapters and can reduce inference latency.

• Design: Takes the same input as the layer's feed-forward network (FFN), computes its bottleneck transformation, and adds the result to the FFN's output.
• Benefit: Avoids extending the critical path depth of the network, as it doesn't add sequential computation.
• Relation: A specific architectural variant of the broader residual adapter concept.

A hypernetwork is a secondary neural network that generates the weights or parameters for a primary (main) model. In PEFT, it is used to dynamically produce the parameters of trainable components (like adapters) conditioned on a task, enabling efficient multi-task learning.

• Function: Maps a task embedding or condition to the full set of adapter weights.
• Efficiency: The hypernetwork itself is small; it generates the adapter parameters on-demand, avoiding storage of separate adapters for every task.
• Use Case: Central to methods that learn intrinsic task vectors, which are fed into the hypernetwork to generate task-specific deltas.

Sparse fine-tuning is a parameter-efficient method that updates only a strategically selected, sparse subset of a model's parameters, leaving the vast majority untouched. The trainable component here is not an added module, but a selected subset of the model's native weights.

• Mechanisms: Includes Diff Pruning (learning a sparse binary mask) and BitFit (updating only bias terms).
• Selection: The sparsity pattern can be learned or predetermined based on saliency metrics.
• Advantage: Maintains the original model architecture perfectly, with zero inference overhead. The 'component' is the set of active, tunable parameters.