LoRA (Low-Rank Adaptation)

LoRA's core innovation is decomposing a weight update (ΔW) into the product of two low-rank matrices, A and B. For a pre-trained weight matrix W ∈ ℝ^(d×k), LoRA constrains ΔW = BA, where B ∈ ℝ^(d×r), A ∈ ℝ^(r×k), and the rank r << min(d, k). This reduces trainable parameters from d×k to r×(d+k). For a 7-billion parameter model with a rank r=8, LoRA may train only ~4 million parameters—a reduction of over 99.9%.

Unlike adapter layers that add sequential computation, LoRA's trained matrices (BA) can be merged additively with the frozen base weights (W) after training: W' = W + BA. This creates a single, standard-weight matrix, resulting in a model that is architecturally identical to the original for inference. This means:

• Zero added latency during deployment.
• No need for specialized runtime logic to handle adapters.
• The merged model can be exported and served like any standard model.

Because each LoRA adaptation is a small set of matrices (often <100MB), multiple specialized adapters can be stored for a single base model. This enables:

• Rapid switching between tasks (e.g., code generation, customer support, creative writing) by loading different adapter weights.
• Modular composition of skills by combining multiple adapters (e.g., a language style adapter + a domain knowledge adapter).
• Efficient A/B testing of different model behaviors without maintaining multiple full-sized model copies.

By drastically reducing the number of gradients and optimizer states that need to be stored in GPU memory, LoRA makes fine-tuning large models accessible. Key advantages include:

• Training on consumer-grade GPUs (e.g., a single 24GB GPU can fine-tune a 65B parameter model).
• Faster training cycles due to smaller optimizer states, especially with memory-efficient optimizers like AdamW 8-bit.
• Lower cloud compute costs, as fine-tuning requires less VRAM and can be completed faster.

Since the vast majority of the pre-trained model's weights are frozen and only the low-rank matrices are updated, the model's general world knowledge and linguistic capabilities are largely preserved. The adaptation is highly targeted, which helps prevent the phenomenon of catastrophic forgetting, where a model loses previously learned information when trained on new data. This makes LoRA particularly suitable for domain adaptation and instruction tuning, where the goal is to add new skills without degrading base performance.

A PEFT method where small, trainable neural network modules are inserted between the layers of a frozen pre-trained model. Unlike LoRA, which adds low-rank matrices in parallel, adapters are sequential bottlenecks that project activations into a lower dimension, process them, and project back. They introduce a small computational overhead during inference.

• Key Distinction: Sequential insertion vs. LoRA's parallel injection.
• Use Case: Historically used in NLP before the widespread adoption of LoRA.

A method that freezes the entire pre-trained model and only optimizes a set of continuous, task-specific vectors (soft prompts) that are prepended to the input embeddings. The model's weights remain unchanged.

• Soft Prompts: Learned vector sequences, as opposed to discrete text prompts.
• Parameter Efficiency: Extremely efficient, often tuning < 1% of a model's parameters.
• Contrast with LoRA: Modifies the input space rather than the internal weight matrices.

An evolution of prompt tuning designed for generative language models. Instead of prepending tunable tokens only to the input layer, prefix tuning prepends trainable vectors to the hidden states at every layer of the transformer architecture. This allows for more expressive, layer-specific conditioning.

• Architecture: Injects tunable parameters into the key and value matrices of the attention mechanism at each layer.
• Relation to LoRA: Both modify internal activations/weights, but Prefix Tuning affects the attention context while LoRA directly adapts weight matrices.

An extension of LoRA that enables fine-tuning of quantized large language models. QLoRA uses a novel 4-bit quantization technique (NormalFloat4) to compress a pretrained model to 4-bit precision. It then trains LoRA adapters on top of this frozen, quantized base model, achieving performance close to full 16-bit fine-tuning.

• Breakthrough: Made fine-tuning 65B parameter models on a single 48GB GPU feasible.
• Core Innovation: Backpropagation through a quantized, frozen model via a custom 4-bit data type and paged optimizers.

A PEFT method that learns task-specific vectors that rescale (inhibit or amplify) the inner activations of a frozen model. These learned scaling vectors are applied to the keys and values in the attention mechanism and to the intermediate activations in feed-forward networks.

• Mechanism: Element-wise multiplication (scaling) rather than matrix addition like LoRA.
• Efficiency: Adds even fewer parameters than standard LoRA, as it learns vectors instead of matrices.

The traditional method of adapting a pre-trained model where all of its parameters are updated during training on a new dataset. This serves as the baseline against which PEFT methods like LoRA are compared.

• Compute Cost: Prohibitively expensive for models with billions of parameters.
• Storage: Requires saving a full copy of the entire updated model for each task.
• Catastrophic Forgetting: High risk of losing general knowledge from the pre-training phase.
• LoRA's Advantage: LoRA mitigates these issues by updating only a tiny fraction of parameters, stored in separate, small adapter files.