LoRA Adapter

LoRA constrains weight updates to a low-rank representation by factorizing the update matrix ΔW into two smaller matrices A and B, where ΔW = BA. For a weight matrix of dimension d×k, LoRA uses matrices of size d×r and r×k, where the rank r is typically between 4 and 64. This reduces trainable parameters by orders of magnitude—a 1-billion-parameter genomic model might require updating only 0.1% of its weights when adapting to a new species' regulatory grammar.

During LoRA fine-tuning, the original model weights are completely frozen and receive no gradient updates. Only the injected low-rank matrices are trained. This preserves the broad genomic knowledge encoded in the foundation model—such as evolutionary conservation patterns and splice site motifs—while allowing efficient specialization. For genomic MLOps pipelines, this means a single base model checkpoint can serve as the immutable backbone for dozens of task-specific LoRA adapters targeting different variant types or species.

LoRA is typically applied to the query, key, value, and output projection matrices within transformer self-attention blocks. In genomic DNA language models like Enformer or HyenaDNA, these linear layers encode the positional and contextual relationships between nucleotide k-mers. By selectively adapting only these attention projections, LoRA efficiently captures domain-specific sequence motifs—such as transcription factor binding sites or chromatin accessibility patterns—without modifying the feed-forward layers that store more general biological knowledge.

After training, the low-rank matrices can be merged back into the original weights via a simple addition: W' = W + BA. This produces a single, unified model checkpoint with zero additional inference latency—a critical advantage for high-throughput genomic inference serving. Unlike adapter layers that add sequential computation, merged LoRA weights execute identically to the original architecture. For continuous batching scenarios in genomic model serving, this means no throughput penalty when deploying adapted models.

LoRA enables hot-swapping of task-specific adapters at inference time without reloading the base model. A single genomic foundation model can dynamically load a variant-calling adapter, a splice-site prediction adapter, or a promoter-region adapter based on the incoming request. This architecture dramatically reduces GPU memory pressure in serving environments—instead of hosting N full model copies, the system maintains one base model and N lightweight adapter matrices, each typically only a few megabytes for genomic-scale models.

Because the pre-trained weights remain frozen, LoRA provides an implicit regularization against catastrophic forgetting. The model cannot overwrite previously learned genomic representations—it can only learn low-rank perturbations. This is particularly valuable when adapting models trained on human reference genomes to non-model organisms or metagenomic samples, where the underlying sequence distribution shifts but fundamental biological patterns must be preserved. The rank constraint acts as a bottleneck that enforces parameter-efficient, generalizable adaptation.

A memory-saving technique that trades compute for memory by discarding intermediate activations during the forward pass and recomputing them during the backward pass. When combined with LoRA, it enables training on extremely long DNA sequences.

• Reduces peak memory by 40-60%
• Only the frozen backbone activations are checkpointed; LoRA adapters remain in memory
• Essential for processing full-length chromosomes or long-read sequencing data

A technique that uses both FP16/BFLOAT16 and FP32 formats during training. The frozen backbone and LoRA adapters can operate at different precisions, with the adapter weights typically kept in higher precision for stability.

• Accelerates genomic model training by 2-3x on modern GPUs
• BFLOAT16 preserves the dynamic range needed for attention scores
• Critical for training on large genomic batch sizes without overflow errors

An alternative parameter-efficient method that prepends learnable virtual tokens to the input sequence rather than modifying internal weights. LoRA generally outperforms prefix tuning on genomic tasks requiring fine-grained sequence understanding.

• Prefix tuning: Modifies attention activations; fewer parameters
• LoRA: Modifies weight matrices; higher fidelity to full fine-tuning
• LoRA preferred for tasks like variant effect prediction requiring precise nucleotide-level reasoning