Parameter-Efficient Fine-Tuning (PEFT)

Low-Rank Adaptation (LoRA) injects trainable rank decomposition matrices into transformer layers while keeping the original pre-trained weights frozen. It hypothesizes that weight updates during adaptation have a low intrinsic rank. Instead of updating the full weight matrix (W \in \mathbb{R}^{d \times k}), LoRA represents the update as (\Delta W = BA), where (B \in \mathbb{R}^{d \times r}) and (A \in \mathbb{R}^{r \times k}) with rank (r \ll \min(d, k)).

• Key Mechanism: The forward pass becomes (h = W_0x + \Delta W x = W_0x + BAx).
• Efficiency: Reduces trainable parameters by orders of magnitude (e.g., ~0.1% of the model).
• Inference: The low-rank matrices can be merged with the base weights for zero-latency overhead.
• Use Case: The standard for fine-tuning large language models like LLaMA and GPT.

Adapter layers are small, bottleneck feed-forward networks inserted sequentially after the multi-head attention and feed-forward modules within a transformer block. The original model parameters remain frozen.

• Architecture: A typical adapter consists of a down-projection matrix, a non-linearity, and an up-projection matrix, returning to the original feature dimension.
• Insertion Points: Placed within the transformer's residual connections, allowing the model to process inputs through the frozen layer and then through the trainable adapter.
• Parameter Efficiency: Adds only ~0.5-8% new parameters per task.
• Trade-off: Introduces a slight sequential inference overhead as data must pass through the extra modules. Used extensively in NLP for multi-task learning.

These methods prepend a sequence of trainable continuous vectors (soft prompts) to the model's input or hidden states, steering the model's behavior without modifying its core weights.

• Prefix-Tuning: Optimizes a sequence of continuous task-specific vectors (the prefix) that are prepended to the keys and values at every layer of the transformer's attention mechanism. The model's own parameters are frozen.
• Prompt Tuning: A simplified version that only prepends trainable vectors to the input embedding layer at the model's outset. It scales well with model size.
• Mechanism: The soft prompts act as a contextual bias, guiding the frozen model's generative or discriminative process for a specific task.
• Advantage: Extremely parameter-efficient (e.g., 0.01% of model size) and allows for multi-task serving by swapping prompts.

(IA)^3 is a PEFT method that scales inner activations of a frozen model using learned task-specific vectors. It introduces three sets of small, learned vectors that perform element-wise multiplication on key intermediate activations.

• Learned Components:
  • Key scaling vector for attention modules.
  • Value scaling vector for attention modules.
  • Feed-Forward scaling vector for intermediate activations.
• Operation: For an activation tensor (h), the method computes (h \odot l), where (l) is the learned scaling vector and (\odot) is element-wise multiplication.
• Efficiency: Adds even fewer parameters than LoRA (e.g., ~0.01% of model size) while maintaining strong performance.
• Use Case: Highly effective for multi-task instruction tuning, as scaling vectors can be composed additively.

QLoRA combines LoRA with 4-bit quantization of the base model, enabling the fine-tuning of extremely large models on a single GPU.

• Core Innovation: Uses 4-bit NormalFloat (NF4) quantization, an information-theoretically optimal data type for normally distributed weights, and Double Quantization to quantize the quantization constants.
• Memory Workflow: The base model weights are stored in 4-bit. During the forward and backward pass, weights are dequantized to 16-bit only for computation using a low-precision storage/high-precision compute pattern.
• PEFT Component: LoRA adapters are applied to the dequantized weights and are themselves stored in 16-bit precision.
• Result: Enables fine-tuning a 65B parameter model on a single 48GB GPU, matching the performance of full 16-bit fine-tuning. It is a cornerstone for accessible LLM research.

This family of techniques identifies and updates only a critical, sparse subset of the model's original parameters, based on importance metrics or architectural principles.

• Diff Pruning: Learns a task-specific sparse diff mask (\Delta) applied to a subset of base weights: (\theta_{task} = \theta_{base} + \delta \odot m), where (m) is a sparse binary mask.
• BitFit: A极端 case of sparsity where only the bias terms within the model are tuned, leaving all weight matrices frozen. Surprisingly effective for many tasks.
• Selection Methods: Parameters can be chosen based on gradient magnitude, Fisher information, or activation-based importance scores from a forward pass on task data.
• Advantage: Maintains the exact original model architecture and inference graph, with no additional modules. Offers a direct, interpretable form of efficiency.

A machine learning paradigm where a model learns a sequence of tasks over time, aiming to accumulate knowledge without catastrophically forgetting previously learned information. PEFT methods are particularly well-suited for this setting because they isolate task-specific knowledge in small, modular parameters (like adapters), which can be stored and reactivated without interfering with the core model.

• Key Challenge: The stability-plasticity dilemma.
• PEFT Advantage: Techniques like task-specific adapters or LoRA modules provide a natural architectural separation that protects previously acquired knowledge.

The tendency of a neural network to abruptly and drastically lose performance on previously learned tasks when trained on new data. This is the core problem continual learning and multi-task PEFT aim to solve.

• Cause in Full Fine-Tuning: Unconstrained gradient updates overwrite weights important for prior tasks.
• Mitigation via PEFT: By freezing the base model and updating only a small set of parameters (e.g., <1% of total weights), the foundational knowledge remains intact. The risk is confined to the small, often task-specific, parameter subset.

A paradigm where a single model is trained simultaneously on multiple related tasks, leveraging shared representations to improve generalization. PEFT transforms this from a joint training problem into a modular composition problem.

• Traditional MTL: All model weights are updated for all tasks, which can lead to negative transfer (interference).
• PEFT for MTL: A shared base model is kept frozen. Each task gets its own small set of trainable parameters (e.g., an adapter, LoRA rank, or prefix). This allows positive forward transfer from the shared base while minimizing inter-task interference.

A core PEFT technique where small, trainable neural network modules (adapters) are inserted into the layers of a frozen pre-trained model. This is a primary method for implementing continual and multi-task learning with PEFT.

• Mechanism: Adapters are typically a down-projection, non-linearity, and up-projection sequence inserted after the feed-forward network in a transformer block.
• For Multi-Task/Continual Learning: A unique adapter can be trained for each task and swapped in at inference time, or a compositional method can blend them. This provides clear parameter isolation for knowledge retention.

A seminal regularization-based continual learning method. It estimates the importance (Fisher information) of each model parameter for previous tasks and penalizes changes to important parameters during new learning.

• Contrast with PEFT: EWC operates on all model parameters but constrains their movement. PEFT methods like LoRA or adapters sidestep this by freezing all original parameters and adding new ones.
• Synergy: Concepts from EWC, like parameter importance, can inform which parts of a model to target with sparse PEFT methods or how to regularize the small set of trainable PEFT parameters.

A rehearsal-based continual learning technique where a model is periodically retrained on a stored subset of data (replay buffer) from previous tasks to mitigate forgetting.

• Challenge: Storing raw data raises privacy and storage concerns.
• PEFT Integration: When combined with PEFT, replay can be far more efficient. Instead of rehearsing to protect the entire massive network, rehearsal only needs to stabilize the small set of task-specific parameters (e.g., a single adapter). Generative replay, using a model to create synthetic past data, can also be more feasible when only fine-tuning a small generator component.