Adapter Weights

Adapter weights represent a tiny fraction of the base model's total parameters. For example, in Low-Rank Adaptation (LoRA), the update to a weight matrix W (of dimension d x k) is approximated as ΔW = B * A, where A and B are low-rank matrices of dimensions d x r and r x k. The rank r is typically << min(d, k). This means training only r*(d + k) parameters instead of the full d*k parameters of the layer. Common ranks (r) range from 4 to 64, often resulting in adapter weights that are less than 1% of the original model's size.

Adapter weights are not added to every layer. They are strategically injected into specific target modules within the network architecture. Common targets include:

• The query and value projection matrices in transformer attention blocks.
• The feed-forward network's up/down projection matrices.
• The output projection of certain layers. This selective injection is based on the empirical finding that adapting these specific subspaces is highly effective for task adaptation, while updating all layers is computationally wasteful. The choice of target modules is a key hyperparameter for adapter-based methods.

A defining characteristic is that the original pre-trained weights of the base model remain entirely frozen and non-trainable during fine-tuning. The adapter weights learn a task-specific delta (ΔW)—the change needed to adapt the frozen function. This offers major advantages:

• Mitigates Catastrophic Forgetting: The core knowledge of the base model is preserved.
• Enables Rapid Task Switching: Multiple lightweight adapter sets can be trained for different tasks and swapped in/out against the same frozen base model.
• Improves Training Stability: The optimization landscape is constrained, often leading to more stable convergence compared to full fine-tuning.

Adapter weights are inherently modular components. A single trained adapter is a self-contained module representing adaptation for one task. This enables powerful composition techniques:

• Task Arithmetic: Adapter weights (deltas) from different tasks can be added or interpolated. For example, Adapter_TaskC ≈ Adapter_TaskA + Adapter_TaskB.
• Model Merging: The trained low-rank matrices can be merged into the base weights for a standalone model: W' = W + ΔW. This eliminates inference overhead.
• Stacking: Some methods allow adapters to be sequentially stacked for multi-task or continual learning. This modularity is foundational for building flexible, multi-purpose model systems.

The primary engineering benefit of adapter weights is dramatic resource savings:

• Memory Efficiency: Only the adapter parameters and their optimizer states (e.g., momentum for Adam) are stored in GPU memory during training, not the full gradients for the base model. This can reduce VRAM usage by 60-80%.
• Compute Efficiency: The forward pass requires a small additional computation: h = Wx + (B*A)x. The backward pass only calculates gradients for the tiny adapter matrices. This leads to faster training steps and lower GPU-hour costs.
• Storage Efficiency: A fine-tuned model is represented as the small adapter file (often a few MBs) plus a pointer to the base model, not a duplicate of all billions of parameters.

In methods like LoRA, adapter weights are not unstructured. They enforce a low-rank bottleneck structure through the two-matrix product B*A. This structure acts as a strong inductive bias and regularizer:

• It hypothesizes that the optimal weight update for task adaptation resides on a low intrinsic dimension manifold.
• The bottleneck prevents the adapter from overfitting to noise in the small fine-tuning dataset.
• The rank r explicitly controls the capacity and expressiveness of the adapter—a higher rank allows for more complex adaptations but uses more parameters. This structured design is key to the method's parameter efficiency and generalization.

Delta weights (ΔW) represent the learned parameter change applied to a frozen pre-trained model. In full fine-tuning, ΔW is a dense matrix of the same size as the original layer. Adapter-based methods like LoRA constrain ΔW to a parameter-efficient form—such as a product of low-rank matrices—dramatically reducing the number of trainable parameters. This delta is added to the frozen base weights during the forward pass: W_final = W_base + ΔW. The concept of a small, learned delta is foundational to modular adaptation.

Parameter efficiency is a core design principle where a model achieves strong performance while updating or adding only a minimal fraction of its total parameters. Adapter weights are the embodiment of this principle. For example, a 7-billion parameter model fine-tuned with LoRA might train only 0.1% of its weights (e.g., 7 million adapter parameters) while matching the performance of full fine-tuning. Key metrics include:

• Trainable Parameter Ratio: Adapter Params / Total Model Params.
• Memory Footprint: The reduction in GPU VRAM required for optimizer states. This efficiency enables adaptation on consumer hardware.

Target modules are the specific layers within a neural network where adapter weights are injected. Not all layers are equally important for adaptation. Common targets in transformer models include:

• Attention Projections: The query (q_proj), key (k_proj), value (v_proj), and output (o_proj) matrices in self-attention blocks.
• Feed-Forward Networks: The up_proj and down_proj layers in the MLP blocks. The selection of target modules is a critical hyperparameter. Injecting adapters into the attention layers is often most effective for task adaptation, as these layers govern contextual relationships. Strategic placement balances performance gains with parameter count.

Merging is the process of analytically combining trained adapter weights with the frozen base model weights to create a single, consolidated model for inference. For LoRA, where ΔW = B * A, merging computes W_merged = W_base + (B * A) * scaling. This eliminates the separate adapter forward pass, resulting in:

• Zero inference latency overhead: The merged model runs at the same speed as the original base model.
• Simplified deployment: A single model file replaces the base model plus adapter checkpoints. Merging is reversible, allowing adapters to be subtracted, which is useful for task arithmetic and model editing.

Catastrophic forgetting is the tendency of a neural network to lose previously learned knowledge when trained on new data. Full fine-tuning, which updates all parameters, is highly susceptible to this. Adapter weights mitigate this risk by making minimal, constrained updates to the base model. The vast majority of original weights remain unchanged, preserving the model's foundational knowledge. This makes PEFT ideal for:

• Sequential task learning: Adapting a model to a new domain without degrading its performance on prior tasks.
• Safe experimentation: Enabling low-risk model customization without corrupting expensive pre-trained assets.

Task arithmetic is a model editing technique that treats adapter weights (or other parameter deltas) as task vectors. These vectors can be linearly combined. For instance, if ΔW_taskA and ΔW_taskB are LoRA adapters for different tasks, a new adapter can be created via ΔW_combined = α * ΔW_taskA + β * ΔW_taskB. This enables:

• Multi-task models: Combining capabilities (e.g., translation and summarization) without additional training.
• Capability interpolation: Adjusting the strength of different skills.
• Negative task vectors: Subtracting a vector to remove undesired behaviors. This demonstrates the modular, composable nature of adapter-based adaptations.