LoRA (Low-Rank Adaptation)

LoRA is grounded in the hypothesis that the change in weights during model adaptation has a low intrinsic rank. Instead of updating a full weight matrix $W \in \mathbb{R}^{d \times k}$, LoRA models the update $\Delta W$ as a product of two smaller matrices $B$ and $A$, where $\Delta W = BA$. If the rank $r$ is much smaller than $d$ and $k$, the number of trainable parameters is drastically reduced from $d \times k$ to $r \times (d + k)$.

A critical architectural advantage of LoRA is that the low-rank matrices can be merged into the frozen pre-trained weights after fine-tuning. Because $W + BA$ is computed once and stored as a single matrix, the final model architecture is identical to the original. This means there is absolutely zero additional computational overhead or latency during inference, unlike adapter layers which introduce sequential bottlenecks.

While LoRA can theoretically be applied to any dense layer, the original implementation focuses exclusively on the self-attention module of the Transformer. Specifically, it adapts the query ($Q$), key ($K$), value ($V$), and output ($O$) projection matrices. This targeted approach is highly effective because adapting the attention mechanism is sufficient to steer the model's behavior for most downstream tasks, leaving the large feed-forward network (FFN) layers untouched.

Because the base model remains frozen and shared, a single pre-trained model can be deployed with hundreds of different LoRA adapters for distinct tasks. Each adapter is typically only a few megabytes in size. The serving system can dynamically load and unload these adapters on a per-request basis, enabling cost-effective multi-task serving without the prohibitive memory cost of loading multiple full copies of a fine-tuned model.

Despite the massive reduction in trainable parameters, LoRA consistently achieves performance on par with or exceeding full fine-tuning across diverse benchmarks. By adjusting the rank $r$, practitioners can trade off between parameter efficiency and model capacity. A higher rank captures more task-specific information, but even very low ranks (e.g., $r=1$ or $r=4$) often yield surprisingly strong results, validating the low intrinsic rank hypothesis.

LoRA adapters trained on distinct tasks can be combined through simple arithmetic operations. For instance, a LoRA adapter trained on a specific style can be added to an adapter trained on a specific domain to create a new model that exhibits both characteristics simultaneously. This composability allows for the creation of novel model behaviors without any additional training, simply by merging the low-rank weight deltas.