Trainable Parameters

Trainable parameters in PEFT represent a sparse subset of the model's total parameter space. Instead of updating all weights (full fine-tuning), methods like BitFit (which updates only bias terms) or sparse fine-tuning selectively activate a tiny fraction—often less than 1%—of the model's parameters. This sparsity is the core mechanism for achieving massive reductions in memory footprint and compute cost during adaptation.

PEFT trainable parameters are typically additive. They introduce new, task-specific parameters (e.g., Adapter modules, LoRA matrices) alongside the frozen base model. The original pre-trained knowledge is preserved intact, preventing catastrophic forgetting. The adaptation is captured in a separate, composable component, often called delta weights or a task vector, which can be applied, removed, or combined.

Many PEFT methods impose a low-rank structure on the trainable parameter updates to maximize efficiency. Low-Rank Adaptation (LoRA) hypothesizes that weight updates during adaptation have a low 'intrinsic rank.' It represents the update ΔW for a weight matrix as the product of two smaller matrices (B * A), where the rank is a critical hyperparameter controlling capacity. This reduces parameters from d * k to r * (d + k), where r << d, k.

Adapter-based methods use a bottleneck architecture to limit trainable parameters. A standard adapter applies a down-projection to a small bottleneck dimension, a non-linearity, and an up-projection back to the original dimension. The bottleneck dimension (e.g., 64) and reduction factor (e.g., 16) determine parameter count. This forces information through a compressed, learnable channel, enabling efficient task-specific transformation of activations.

Trainable parameters are inserted at precise injection points within the model architecture. Common locations include:

• After the multi-head attention module.
• After the feed-forward network in a transformer block.
• As prefixes to the key/value matrices in attention (Prefix Tuning). The choice of injection point (e.g., for BERT Adapters or ViT Adapters) is critical for effectively steering model behavior with minimal parameters.

The core principle is modality-agnostic: the same PEFT concepts apply to text (LLMs), vision (ViTs), audio, and multimodal models. However, specialized instantiations exist:

• Visual Adapters for image tasks.
• Audio Adapters for speech models.
• VL-Adapters or Cross-Modal Adapters for vision-language models like CLIP, which adapt the fusion mechanisms between modalities using efficient parameters.

The frozen backbone is the large, pre-trained base model (e.g., BERT, ViT, GPT) whose original parameters are kept fixed during parameter-efficient fine-tuning. This is the foundation upon which PEFT operates, preserving the model's general knowledge and preventing catastrophic forgetting. Only the small set of newly introduced trainable parameters (like adapters or prompts) are updated, drastically reducing memory and compute requirements compared to full model fine-tuning.

Delta weights (ΔW) refer to the small, learned parameter changes applied to a frozen pre-trained model during PEFT. They mathematically represent the task-specific adaptation. In methods like LoRA, these deltas are approximated by low-rank matrices. Key properties include:

• Additive Application: The adapted weights are computed as W + ΔW.
• Encapsulated Knowledge: The delta contains all information learned for the new task.
• Modularity: Deltas from different tasks can be stored, combined, or subtracted, enabling model merging and task arithmetic.

Injection points are the specific architectural locations within a neural network where parameter-efficient modules are inserted to introduce trainable parameters. Strategic placement is critical for performance. Common points in transformers include:

• After the Multi-Head Attention block.
• After the Feed-Forward Network block.
• Within attention key/value projections (for prefix tuning). The choice of injection point determines how the adapted signals flow through the frozen backbone and influences the module's impact on model behavior.

In adapter-based methods, the bottleneck dimension is the size of the hidden layer within the adapter module. It is the primary hyperparameter controlling the adapter's capacity and number of trainable parameters. The adapter projects the input activation down to this dimension, applies a non-linearity, then projects back up. A smaller bottleneck increases efficiency but may reduce representational power. It is often set via a reduction factor (e.g., reducing a 768-dim activation to 96, a factor of 8).

In Low-Rank Adaptation (LoRA), the rank (r) is the intrinsic dimension of the low-rank matrices used to approximate the weight update ΔW = BA. It is the central hyperparameter governing the number of trainable parameters. A higher rank increases adaptability at the cost of more parameters. For a weight matrix of dimension d x k, the trainable parameter count is r*(d+k). Typical ranks are very low (e.g., 4, 8, 16), which is sufficient because weight updates during fine-tuning are hypothesized to have a low intrinsic rank.

A task vector is the arithmetic difference between the weights of a fully fine-tuned model and its pre-trained base, τ = θ_fine-tuned - θ_base. In PEFT, the delta weights effectively form a sparse or low-rank task vector. This conceptualization enables powerful operations:

• Model Merging: Adding task vectors from multiple models: θ_merged = θ_base + ατ_A + βτ_B.
• Task Negation: Forgetting a skill by subtracting its vector.
• Linearity Exploration: Studying how model capabilities change along vector directions.