Prefix Tuning

Unlike discrete text prompts, prefix tuning learns a sequence of continuous, task-specific embedding vectors. These vectors are prepended to the keys and values of the transformer's multi-head attention mechanism at every layer. This allows the model to be conditioned on a rich, learned representation that is optimized via gradient descent, providing more expressive control than hand-crafted text prompts.

The core innovation is that the original pre-trained model parameters remain completely frozen during fine-tuning. Only the newly added prefix vectors are updated. This makes the method highly parameter-efficient, as it updates a tiny fraction (often < 1%) of the total model parameters. It preserves the model's general knowledge while adapting its behavior for a specific task.

The prefix is injected into the attention computation. For each layer l and head h, the original key K and value V matrices are concatenated with the trainable prefix matrices P_K^l and P_V^l:

• K' = [P_K^l; K]
• V' = [P_V^l; V] This modifies the attention context for all subsequent tokens, steering the model's generation without altering its core weights. The prefix length is a hyperparameter controlling capacity.

Prefix tuning is designed for scalability to large models. Since only the prefix parameters are trained, the memory and storage overhead is minimal. For a model with billions of parameters, a prefix of length 20 might add only ~200,000 trainable parameters. This enables fine-tuning of massive models (e.g., GPT-3, T5) on single GPUs, making it practical for enterprise adaptation.

Learned prefixes can exhibit strong transfer learning capabilities. A prefix trained on one task can sometimes be effectively applied to a related task, or prefixes from multiple tasks can be ensembled. Furthermore, because the base model is unchanged, a single model instance can host multiple task-specific prefixes, enabling efficient multi-task serving from one deployed checkpoint.

Vs. Adapters: Adapters insert small feed-forward networks between transformer layers. Prefix tuning operates within the attention mechanism itself, offering a different form of conditioning. Vs. LoRA: LoRA injects low-rank matrices via addition to the weight matrices (W + ΔW). Prefix tuning adds parameters to the activations (keys/values) in the forward pass, not the weights. Both are highly parameter-efficient but modify the model through different mechanisms.

A direct precursor to prefix tuning, prompt tuning learns a small set of continuous embedding vectors (soft prompts) that are prepended to the input sequence. Unlike hard, text-based prompts, these are continuous, trainable parameters optimized via gradient descent. The model's core weights remain entirely frozen.

• Key Difference: Prompt tuning typically adds embeddings only to the input layer, while prefix tuning injects vectors into the attention mechanism of every transformer layer.

P-Tuning is a method for optimizing continuous prompt embeddings, similar to prompt tuning, but it introduces a lightweight LSTM or MLP prompt encoder to generate the continuous prompt tokens. This structure helps model the dependencies between prompt tokens, often leading to more stable optimization and better performance on complex reasoning tasks compared to directly optimizing standalone embeddings.

Adapter layers are small, bottleneck feed-forward networks (typically two linear layers with a non-linearity) inserted in parallel or sequentially within transformer blocks. During fine-tuning, only the adapter parameters are updated while the original model is frozen.

• Architecture: Places a compact module (e.g., down-project → ReLU → up-project) within each transformer block.
• Contrast with Prefix Tuning: Adapters modify the feed-forward pathway, while prefix tuning modifies the attention key-value memory.

LoRA freezes the pre-trained weights and injects trainable low-rank decomposition matrices into transformer layers. For a weight matrix W, LoRA represents its update as ΔW = BA, where B and A are low-rank matrices. This update is added to the frozen W during the forward pass.

• Parameter Efficiency: Achieves similar efficiency to prefix tuning but operates via additive low-rank updates to weight matrices rather than prepending to activations.
• Deployment Advantage: The low-rank matrices can be merged with the base weights post-training for zero-inference overhead.

Delta tuning is an umbrella term for the family of parameter-efficient fine-tuning methods that update only a small subset of parameters (the delta or change) relative to the pre-trained model. Prefix tuning, LoRA, and adapters are all specific instantiations of delta tuning.

• Core Principle: The final model weights are expressed as W_final = W_pretrained + ΔW, where ΔW is sparse or low-rank.
• Unified View: This framework groups methods by how they parameterize and apply the delta to the frozen base model.

An extremely lightweight method, BitFit proposes fine-tuning only the bias terms within a transformer model. All weight matrices (e.g., in attention and feed-forward layers) remain frozen.

• Extreme Sparsity: Updates <1% of total parameters in models like BERT.
• Mechanism: Shows that bias terms capture significant task-specific adaptation signals. It provides a strong baseline, demonstrating that not all parameters are equally important for adaptation.
• Comparison: Represents the minimal end of the PEFT spectrum, whereas prefix tuning updates a small but dedicated set of new parameters.