Low-Rank Adaptation (LoRA)

LoRA's core innovation is its parameter efficiency. Instead of fine-tuning all weights in a dense layer (e.g., a 4096x4096 matrix with ~16.7M parameters), LoRA injects two much smaller matrices. For a rank r, the trainable parameters are reduced from d * k to r * (d + k). For a typical rank of 8, this results in a reduction of 100x to 1000x in trainable parameters compared to full fine-tuning. This makes adaptation feasible on consumer-grade GPUs.

A key advantage is zero-overhead deployment. After training, the low-rank matrices B and A can be merged with the original frozen weights W through a simple addition: W' = W + BA. This creates a single, dense weight matrix identical in structure to the original model. The merged model has no additional computational graph complexity and runs at the exact same inference speed as the base model, eliminating the runtime cost of conditional adapters.

LoRA enables modular task specialization. Multiple independent LoRA modules (e.g., for coding, legal analysis, and creative writing) can be trained on a single base model. These modules are stored as small checkpoint files (often just a few MBs). At inference time, they can be:

• Swapped dynamically without reloading the base model.
• Linearly combined (e.g., 0.7 * LoRA_A + 0.3 * LoRA_B) for blended task performance.
• Selectively applied to specific layers or model components for targeted adaptation.

By keeping the original pre-trained weights W frozen, LoRA preserves the model's general knowledge. The adaptation is achieved through a low-rank update ΔW = BA, which represents a constrained direction in the weight space. This approach minimizes the risk of catastrophic forgetting—where the model loses its previously learned capabilities—because the foundational representations remain intact. The model retains its performance on broad tasks while gaining new, specialized skills.

LoRA dramatically lowers the computational and financial barrier to fine-tuning. Key efficiencies include:

• Reduced GPU Memory: Only the gradients for the LoRA matrices and a subset of optimizer states need to be stored, enabling fine-tuning of billion-parameter models on GPUs with 16-24GB VRAM.
• Faster Training Cycles: With far fewer parameters to optimize, training converges in hours instead of days.
• Minimal Storage: A full model checkpoint might be 100GB, while a LoRA module is typically 10-200MB, simplifying version control and distribution.

Small, trainable neural network modules (typically two feed-forward layers with a non-linearity) inserted sequentially between the layers of a frozen pre-trained model. Unlike LoRA's additive low-rank update, adapters create a sequential bottleneck, adding a small amount of latency. They were a precursor to LoRA, establishing the core PEFT paradigm of freezing the base model and tuning only injected parameters.

• Key Difference: Sequential vs. parallel weight modification.
• Common Structure: Down-projection → Non-linearity (e.g., ReLU) → Up-projection.
• Primary Use Case: Early PEFT method for task adaptation in NLP.

A process where a model is fine-tuned with simulated quantization noise (rounding/clamping) in the forward pass. This allows the model weights—including those of LoRA matrices—to adapt to the lower precision (e.g., INT8, INT4) that will be used during deployment. Combining QAT with LoRA is a powerful strategy for producing ultra-efficient, adapted models for edge inference.

• Synergy with LoRA: Train LoRA adapters with quantization simulation for robust on-device deployment.
• Objective: Minimizes accuracy drop from post-training quantization.
• Output: A model whose parameters are optimized for integer arithmetic.

A neural network architecture where different parts of the model (the 'experts') are activated sparsely per input. While not a fine-tuning method itself, MoE relates to LoRA through the theme of conditional computation and parameter efficiency. Sparse MoE models have a large parameter count but a small active parameter count per token, similar to how LoRA keeps the base model frozen and only uses a small set of active adapters.

• Shared Principle: Achieving high model capacity with sub-linear computational cost.
• Contrast: MoE is an architectural choice for pre-training; LoRA is a fine-tuning method for adaptation.
• Example: Models like Mixtral 8x7B use Sparse MoE.

A PEFT method that prepends a sequence of continuous, trainable 'virtual token' vectors to the input sequence at each transformer layer. These prefix vectors steer the model's attention and activation patterns for a new task. Unlike LoRA, which modifies weight matrices directly, Prefix Tuning operates in the activation space, leaving all model weights untouched.

• Mechanism: Learns optimal [key, value] activations to prepend to the sequence context.
• Efficiency: Trainable parameters scale with layer count and prefix length, not model width.
• Trade-off: Can increase sequence length and inference memory, unlike weight-based methods like LoRA.

An extension of LoRA that enables fine-tuning of quantized models. QLoRA uses a 4-bit quantized pre-trained model (frozen) and trains LoRA adapters in a higher precision (e.g., BrainFloat 16). The gradients are backpropagated through the quantized weights via a novel technique, enabling memory-efficient fine-tuning of extremely large models (e.g., 65B parameter models on a single 48GB GPU).

• Core Innovation: Backpropagation through 4-bit quantized weights via NormalFloat 4 quantization and double quantization.
• Memory Reduction: Drastically reduces the memory footprint of the base model during training.
• Result: Makes large-model fine-tuning accessible with consumer-grade hardware.

A PEFT method that introduces trainable scaling vectors ('lora vectors') to multiply (inhibit/amplify) activations within a model, rather than adding a low-rank update to weights like LoRA. These small vectors are applied to the keys, values, and intermediate feed-forward layer activations. It has even fewer parameters than standard LoRA and can be viewed as a simplified, multiplicative form of adaptation.

• Parameter Count: Often fewer trainable parameters than LoRA (scaling vectors vs. matrices).
• Operation: Element-wise multiplication of activations, not matrix addition.
• Advantage: Extremely lightweight and simple to implement alongside other methods.