LoRA (Low-Rank Adaptation)

LoRA is grounded in the hypothesis that the change in weights during model adaptation has a low intrinsic rank. Instead of updating a full weight matrix W, LoRA represents the update as a product of two smaller matrices, A and B, such that the update is ΔW = BA. By keeping the original W frozen and only training A and B, the number of trainable parameters is drastically reduced, often by a factor of 10,000x, without sacrificing the model's original knowledge.

The adaptation is injected directly into the attention mechanism of a transformer model. For a pre-trained weight matrix W₀ ∈ ℝ^(d×k), the forward pass is modified to h = W₀x + BAx, where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with the rank r being much smaller than d and k. Matrix A is typically initialized with a random Gaussian distribution, while B is initialized to zero, ensuring the fine-tuning starts from the pre-trained model's baseline behavior.

Because the original weights are frozen, multiple distinct LoRA adapters can be trained for different tasks and swapped in and out of the base model without any inference latency overhead. This enables a single, large foundation model to be dynamically specialized for:

• Domain-specific tasks: Legal, medical, or financial analysis
• Style transfer: Adapting tone and format
• Language adaptation: Extending to new languages The lightweight adapters (often just a few megabytes) can be stored and loaded on demand, a critical feature for sovereign, multi-tenant deployments.

LoRA eliminates the need to calculate gradients or maintain optimizer states for the vast majority of the model's parameters. This leads to a dramatic reduction in GPU memory footprint during training. For example, fine-tuning GPT-3 175B with Adam requires storing optimizer states that are 2x the model size. With LoRA, the optimizer states are only needed for the tiny injected matrices, reducing the total memory requirement by up to 3x and enabling fine-tuning of massive models on a single GPU, which is essential for air-gapped and on-premises environments.

A key practical advantage of LoRA is that the trained adapter matrices BA can be merged directly into the frozen pre-trained weight matrix W before deployment. The simple linear algebra operation W = W₀ + BA produces a standard model file with no additional computational overhead during inference. This means there is zero added latency compared to the original model, a critical requirement for production serving systems where every millisecond counts.

QLoRA extends the LoRA methodology by backpropagating gradients through a 4-bit NormalFloat quantized pre-trained model. This innovation allows fine-tuning of a 65B parameter model on a single 48GB GPU while preserving full 16-bit task performance. Key techniques include:

• 4-bit NormalFloat: An information-theoretically optimal quantization data type
• Double Quantization: Quantizing the quantization constants to save additional memory
• Paged Optimizers: Using unified memory to handle gradient checkpointing spikes This makes sovereign fine-tuning of state-of-the-art models economically viable on consumer-grade hardware.

A compression technique where a smaller student model is trained to replicate the output distribution of a larger teacher model. Unlike LoRA which adapts an existing model, distillation creates an entirely new, compact architecture.

• Response-based: Student mimics teacher logits
• Feature-based: Student replicates intermediate representations
• Relation-based: Student preserves inter-layer relationships
• Often combined with LoRA adapters for domain-specific teachers

A structural compression method that removes redundant weights or entire neurons from a network. Unstructured pruning zeroes out individual weights, while structured pruning removes entire channels or attention heads.

• Magnitude pruning: Removes smallest absolute weights
• Movement pruning: Considers weight changes during fine-tuning
• LoRA can be applied post-pruning to recover lost accuracy
• Complements quantization for compound compression

Full fine-tuning updates all model parameters, requiring distributed GPU clusters for large models. Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA update <1% of parameters while freezing the backbone.

• Full fine-tuning: catastrophic forgetting risk, high compute cost
• PEFT: modular adapters, rapid task switching
• LoRA adds <1% inference latency when merged with weights
• Ideal for multi-tenant enterprise deployments

Alternative PEFT methods that prepend learnable continuous vectors to the input. Prefix tuning adds virtual tokens to each transformer layer, while prompt tuning adds them only to the input embedding.

• LoRA outperforms both on complex reasoning tasks
• Prompt tuning struggles with small models (<10B params)
• Prefix tuning consumes more inference memory than LoRA
• LoRA's weight-space adaptation generalizes better

The original PEFT approach that inserts small bottleneck neural networks between transformer layers. Each adapter contains a down-projection, non-linearity, and up-projection.

• Introduces additional inference latency due to sequential processing
• LoRA eliminates this latency via weight merging
• Adapters remain useful for multi-task switching without merging
• LoRA's low-rank decomposition is more parameter-efficient