LoRA (Low-Rank Adaptation)

LoRA achieves parameter-efficient fine-tuning by freezing the original, pre-trained model weights and injecting trainable low-rank matrices into specific layers, typically the attention mechanisms. This drastically reduces the number of trainable parameters—often by over 90% compared to full fine-tuning.

• Key Mechanism: Decomposes the weight update (ΔW) for a layer into two smaller matrices: ΔW = B * A, where B and A are low-rank.
• Result: A full 7-billion parameter model might require training only 4-8 million LoRA parameters, enabling fine-tuning on consumer-grade GPUs.

By avoiding updates to the massive pre-trained weight matrices, LoRA significantly lowers GPU memory consumption and computational cost during training. This is because:

• Frozen Base Model: The original weights are kept in a stable, quantized state, requiring only forward passes.
• Small Gradients: Only the gradients for the small injected matrices (A and B) need to be computed and stored.
• Practical Impact: Enables fine-tuning of large models (e.g., 70B parameters) on hardware previously capable of only inference, slashing cloud compute costs.

A critical advantage of LoRA is the elimination of added inference latency. Once training is complete, the low-rank matrices (B and A) can be merged with the original frozen weights (W) through a simple addition: W' = W + BA.

• Merge Operation: This creates a single, consolidated model identical in architecture and size to the original.
• Runtime Performance: The merged model runs at the same speed as the base model, with no extra computational steps or memory overhead during deployment.
• Deployment Flexibility: Allows for storing and switching between multiple adapted models (tasks) as discrete sets of small LoRA weights, merging them on-demand.

LoRA enables a modular approach to model adaptation. Each fine-tuned task is represented by a small, separate set of LoRA weights.

• Task Agility: A single base model can host multiple task-specific adapters. Switching tasks involves swapping in a different set of LoRA matrices.
• Composition: Adapters for different capabilities (e.g., coding, summarization) can potentially be composed or combined.
• Storage Efficiency: Storing many small LoRA files (often <100MB) is far more efficient than storing multiple full-model copies (tens of GBs).

Because the core pre-trained weights remain frozen, LoRA inherently preserves the general knowledge acquired during the model's initial large-scale pre-training. This mitigates catastrophic forgetting, a common problem in full fine-tuning where a model loses its broad capabilities while over-specializing on a new, narrow dataset.

• Stable Foundation: The model retains its original linguistic and reasoning abilities.
• Targeted Adaptation: The low-rank updates provide a focused, minimal adjustment to steer the model for the new task without corrupting its foundational representations.

LoRA has inspired and integrates with several advanced parameter-efficient fine-tuning techniques:

• QLoRA: Combines LoRA with 4-bit quantization of the base model, reducing memory requirements further to fine-tune extremely large models on a single GPU.
• Adapter Layers: A related family of methods that insert small bottleneck modules between transformer layers, but LoRA is often more efficient and performant.
• DoRA (Weight-Decomposed Low-Rank Adaptation): A recent enhancement that decomposes pre-trained weights into magnitude and direction components, applying LoRA only to the directional part for improved learning capacity.

Adapter layers are small, trainable neural network modules inserted between the fixed layers of a pre-trained model. They act as a bottleneck, projecting activations to a lower dimension, applying a non-linearity, and projecting back. This allows for task-specific adaptation with minimal new parameters.

• Key Difference from LoRA: Adapters add sequential computational overhead, while LoRA's low-rank matrices work in parallel, often preserving inference speed.
• Use Case: Historically used in NLP for cross-lingual adaptation before the widespread adoption of transformer-based PEFT methods.

Prompt tuning is a PEFT method that freezes the entire pre-trained model and only learns a set of continuous, task-specific vectors (soft prompts) that are prepended to the input embeddings.

• Mechanism: The model's context window is extended with these trainable embeddings, which steer the model's behavior without changing its weights.
• Efficiency: Extremely parameter-efficient, often using just 0.01% of the model's parameters. However, it can require large batch sizes and many training steps to converge effectively.
• Contrast with LoRA: Modifies the input space rather than the model's internal weights.

QLoRA is an extension of LoRA that enables fine-tuning of quantized large language models. It uses a 4-bit NormalFloat quantization for the base model weights, while the injected LoRA adapters are kept in a higher precision (typically BF16).

• Breakthrough: Allows fine-tuning of 65B parameter models on a single 48GB GPU by drastically reducing memory footprint.
• Backpropagation: Gradients are computed with respect to the quantized weights via a custom backward pass, enabling effective training despite the low-precision storage.
• Significance: Democratizes access to fine-tuning the largest models by making it cost-feasible on consumer hardware.

Prefix tuning is a precursor to prompt tuning, designed for generative language models. Instead of tuning input embeddings, it optimizes a sequence of continuous vectors (the prefix) that are prepended to the keys and values of every attention layer in the transformer.

• Architecture: The learned prefix affects the attention context throughout the model's depth, offering more expressive control than input-level prompts.
• Implementation: Often reparameterized via a small MLP to improve training stability.
• Relation to LoRA: Both modify the model's internal activations (keys/values for prefix, weight deltas for LoRA) but through fundamentally different mechanisms.

IA³ is a PEFT method that scales inner activations of a pre-trained model using learned vectors. It introduces three sets of learned vectors that rescale the keys and values in attention layers and the intermediate activations in feed-forward networks.

• Mechanism: These learned scaling factors are element-wise multipliers, making them even more parameter-light than standard adapters or LoRA.
• Efficiency vs. Performance: Designed for extreme efficiency, often outperforming LoRA on some benchmarks with fewer parameters by directly modulating information flow rather than adding new weight matrices.
• Design Philosophy: Focuses on efficient modulation of existing features rather than addition of new computational paths.

DoRA is an enhancement of LoRA that decomposes a pre-trained weight matrix into magnitude and direction components. It fine-tunes the direction using low-rank matrices (like LoRA) while separately learning a magnitude vector.

• Core Innovation: Decouples the learning of weight direction (handled by low-rank update) and norm (handled by a separate vector). This provides finer-grained control and often leads to performance closer to full fine-tuning.
• Advantage over LoRA: The magnitude vector allows DoRA to more effectively adjust the intensity of feature updates, leading to improved convergence and final task performance without a significant parameter increase.
• Result: Bridges the performance gap between LoRA and full parameter fine-tuning.