LoRA (Low-Rank Adaptation)

LoRA constrains the weight update ΔW to a low-rank representation by factorizing it into two smaller matrices A and B, where ΔW = BA. For a weight matrix of dimension d×k, LoRA trains matrices of size d×r and r×k, where the rank r is typically 1 to 16—orders of magnitude smaller than the original dimensions. This decomposition exploits the hypothesis that the change in weights during fine-tuning has a low intrinsic rank, meaning most of the adaptation signal lives in a small subspace. In genomic models like DNABERT or the Nucleotide Transformer, this allows adapting a 110M-parameter model to a new assay with only 0.1-1% additional trainable parameters.

During LoRA fine-tuning, the original pre-trained weights of the genomic language model remain completely frozen and receive no gradient updates. Only the injected low-rank matrices A and B are trained. This preserves the rich, generalizable representations of regulatory syntax learned during pre-training on massive genomic corpora—such as the human reference genome or multi-species datasets—while allowing efficient specialization. The frozen backbone eliminates the risk of catastrophic forgetting, ensuring the model retains its broad understanding of promoter structures, splice sites, and enhancer grammar even as it adapts to a narrow downstream task like predicting chromatin accessibility in a specific cell type.

LoRA introduces a scaling factor α that controls the magnitude of the low-rank update relative to the original weights. The adapted weight becomes W + (α/r)BA, where r is the rank. This hyperparameter decouples the learning rate from the update magnitude, allowing practitioners to tune the influence of the adaptation without changing the optimizer configuration. In genomic fine-tuning scenarios—such as adapting a foundation model to predict transcription factor binding from ChIP-seq data—adjusting α provides a direct lever to balance pre-trained knowledge retention against task-specific specialization, preventing the low-rank update from overwhelming the frozen representations.

LoRA is typically applied exclusively to the query (Q) and value (V) projection matrices within the multi-head self-attention layers of the transformer architecture. This design choice targets the parameters most responsible for learning contextual relationships between tokens—critical for genomic models that must capture long-range interactions between distal regulatory elements. By leaving the feed-forward layers untouched, LoRA preserves the model's token-level representations while adapting only the attention patterns. For a genomic transformer with 12 layers and 12 attention heads, applying LoRA to Q and V matrices with rank r=8 reduces trainable parameters from 110M to approximately 294K.

Because LoRA adapters are small, self-contained matrices, a single frozen genomic foundation model can be efficiently deployed across multiple downstream tasks without duplicating the base model. Each task—such as variant effect prediction, promoter strength estimation, or splice site detection—receives its own set of LoRA weights stored as separate files typically under 10MB each. At inference time, the appropriate adapter is loaded and merged with the frozen backbone via W + BA, or kept separate for dynamic switching. This architecture enables a single GPU to serve dozens of specialized genomic predictors simultaneously, dramatically reducing infrastructure costs for multi-assay platforms.

Unlike adapter layers that insert additional sequential modules into the network, LoRA's low-rank matrices can be merged directly into the original weight matrix after training via simple addition: W_merged = W + BA. This means the adapted model has the exact same architecture and parameter count as the original during inference—there is no additional computational overhead, no extra layers to execute, and no increase in latency. For latency-sensitive genomic applications such as real-time variant calling during sequencing runs or clinical decision support systems, this property is critical. The model runs at identical speed to the pre-trained base while delivering task-specific accuracy.

The traditional adaptation baseline that LoRA seeks to replace. Full fine-tuning updates all pre-trained weights of a genomic language model during adaptation to a new assay or species.

• Memory Cost: Requires storing a full model copy and optimizer states (e.g., Adam) for each downstream task, often exceeding 100GB+ for a 10B parameter model.
• Catastrophic Forgetting: Risk of overwriting general genomic syntax learned during pre-training when adapting to small, specialized datasets.
• Comparison to LoRA: LoRA reduces trainable parameters by 10,000x while often matching or exceeding full fine-tuning performance on tasks like promoter prediction.

A parameter-efficient alternative that prepends a sequence of continuous, learnable vectors (a 'soft prompt') to the input of a frozen transformer. Unlike LoRA, which modifies internal attention weights, prefix tuning operates entirely at the input level.

• Mechanism: Virtual tokens are concatenated to keys and values in every attention layer, steering the model's behavior without touching its core weights.
• Genomic Application: Effective for adapting DNABERT to new species by learning species-specific prefix vectors that condition the model's attention on the target genome's statistical properties.
• Trade-off: Less expressive than LoRA for tasks requiring fine-grained regulatory element detection, as it cannot modify internal feature detectors.

Small bottleneck neural networks inserted between the existing layers of a frozen pre-trained model. Each adapter consists of a down-projection to a lower dimension, a non-linearity, and an up-projection back to the original dimension.

• Inference Latency: Unlike LoRA, which can be merged into the original weights for zero-latency inference, adapters add sequential computation that increases latency by 5-30%.
• Genomic Use Case: Used in Enformer adaptations where small adapter modules are inserted after each convolutional block to specialize the model for predicting tissue-specific chromatin accessibility tracks.
• LoRA Advantage: LoRA's low-rank matrices can be mathematically fused with frozen weights, eliminating inference overhead entirely.

An extension that combines LoRA with 4-bit NormalFloat quantization of the frozen base model, enabling fine-tuning of a 65B parameter genomic language model on a single 48GB GPU.

• Double Quantization: Quantizes the quantization constants themselves to further reduce memory footprint.
• Paged Optimizers: Uses unified memory paging to handle gradient checkpointing spikes, preventing out-of-memory errors during backpropagation.
• Genomic Relevance: Critical for adapting massive models like the Nucleotide Transformer to rare disease cohorts where only limited computational resources are available at clinical institutions.

The mathematical foundation of LoRA, based on the hypothesis that weight updates during fine-tuning have a low intrinsic rank. A weight update matrix ΔW is factorized into two smaller matrices A and B.

• Dimensions: For a weight matrix W ∈ R^(d×k), LoRA learns B ∈ R^(d×r) and A ∈ R^(r×k) where the rank r << min(d,k). A typical genomic setting uses r=8 or r=16.
• Initialization: A is initialized with random Gaussian values, while B is initialized to zero, ensuring the update starts as an identity function.
• Scaling: The product BA is scaled by α/r, where α is a hyperparameter controlling the magnitude of the adaptation relative to the frozen weights.

A matrix factorization technique that decomposes any matrix into UΣV^T, revealing its rank structure. SVD is used to initialize LoRA matrices and to analyze the rank dynamics of weight updates.

• Initialization Strategy: Some methods initialize LoRA's A and B matrices using the top-r singular vectors of the pre-trained weight matrix, providing a stronger starting point than random initialization.
• Rank Analysis: Post-hoc SVD of the learned ΔW reveals the effective rank required for genomic tasks—promoter prediction often requires higher rank (r=16) than intron-exon boundary detection (r=4).
• AdaLoRA: An adaptive variant that uses SVD to dynamically allocate rank across layers, pruning redundant singular values during training.