Parameter-Efficient Fine-Tuning (PEFT)

LoRA freezes the pre-trained model weights and injects trainable rank decomposition matrices into the attention layers. By representing weight updates as the product of two low-rank matrices, LoRA reduces the number of trainable parameters by 10,000x compared to full fine-tuning while maintaining model quality. The rank r is a critical hyperparameter—typical values range from 4 to 64 for genomic models—balancing adaptation capacity against parameter efficiency. LoRA adds no inference latency because the learned matrices can be merged directly into the original weights.

Adapter modules are small bottleneck neural networks inserted between the layers of a frozen pre-trained model. Each adapter consists of a down-projection to a lower dimension, a non-linear activation, and an up-projection back to the original dimension. During fine-tuning, only these lightweight modules are updated. For genomic language models like DNABERT or Enformer, adapters enable task-specific specialization—such as cell-type-specific expression prediction—without modifying the foundational sequence representations learned during pre-training.

Prefix tuning prepends a sequence of learnable continuous vectors—virtual tokens—to the input of each transformer layer. These prefix vectors steer the frozen model's behavior toward a specific downstream task without modifying any original weights. Unlike prompt engineering, which uses discrete tokens, prefix tuning optimizes continuous embeddings in a differentiable manner. This approach is particularly effective for generative genomic tasks, such as de novo protein design or sequence generation conditioned on desired functional properties.

Prompt tuning learns a small set of task-specific continuous vectors appended to the model's input embedding sequence. The entire pre-trained model remains frozen; only these soft prompts are updated during training. For genomic applications, prompt tuning enables rapid adaptation across multiple downstream tasks—such as variant effect prediction, enhancer-gene linking, and motif discovery—using a single shared backbone model. Each task requires storing only a few kilobytes of prompt parameters rather than a full model copy.

Selective fine-tuning updates only a carefully chosen subset of the pre-trained model's existing parameters while freezing the rest. Selection strategies include:

• Bias-only tuning: updating only bias terms across all layers
• Layer-wise tuning: fine-tuning only the final few transformer blocks
• Attention-head pruning: updating only heads identified as task-relevant via importance scoring In genomic models, this approach can target layers known to capture specific biological features—such as attention heads that learn transcription factor binding motifs—for highly efficient domain adaptation.

QLoRA combines 4-bit weight quantization with Low-Rank Adaptation to enable fine-tuning of massive genomic models on consumer-grade hardware. The pre-trained model is quantized to NF4 (NormalFloat4) precision, dramatically reducing memory footprint, while LoRA adapters are trained in full precision. Key innovations include double quantization to compress scaling constants and a paged optimizer to handle memory spikes. This enables fine-tuning a 65-billion-parameter model on a single GPU, making large-scale genomic foundation model adaptation accessible to academic labs.

The most widely adopted PEFT method that injects trainable low-rank decomposition matrices into the attention layers of a frozen pre-trained model. Instead of updating the original weight matrix W, LoRA learns a low-rank update ΔW = BA, where B and A are small matrices whose product has the same dimensions as W but with drastically fewer parameters. For genomic models like DNABERT or Enformer, LoRA typically reduces trainable parameters by 99.9% while maintaining over 98% of full fine-tuning performance on tasks like promoter identification and variant effect prediction. The rank r (commonly 8-64) controls the trade-off between capacity and efficiency.

Small bottleneck neural networks inserted between the existing layers of a frozen pre-trained transformer. Each adapter consists of a down-projection to a lower dimension, a non-linear activation (typically GELU or ReLU), and an up-projection back to the original dimension, with a residual skip connection. In genomic applications, adapters are placed after the multi-head attention and feed-forward sub-layers of each transformer block. Unlike LoRA, adapters introduce a small inference latency penalty due to the additional sequential computation, but they offer modularity—different adapters can be trained for different cell types or tissues and swapped without modifying the base model.

A PEFT technique that prepends a sequence of learnable continuous vectors (virtual tokens) to the input or to the keys and values at each transformer layer. These prefix vectors are optimized during fine-tuning while the entire pre-trained model remains frozen. In genomic language models, prefix tuning effectively conditions the model on task-specific context—for example, prepending a learned prefix that biases the model toward enhancer-specific patterns versus promoter-specific patterns. The method is particularly parameter-efficient, often requiring fewer than 0.1% of the model's parameters, but can be less expressive than LoRA for complex sequence-to-sequence tasks like gene expression prediction.

A simplified variant of prefix tuning where learnable soft prompts are prepended only to the input embedding layer rather than at every transformer block. This makes prompt tuning extremely parameter-efficient—often just a few thousand parameters total—but limits its capacity to influence deeper model representations. For genomic foundation models, prompt tuning works well on classification tasks like splice site prediction or chromatin state annotation, where shallow conditioning is sufficient. It struggles with tasks requiring deep contextual modulation, such as enhancer-gene linking across long genomic distances. The learned prompt embeddings can be interpreted as a compressed task specification.

An ultra-lightweight PEFT method that rescales the keys, values, and feed-forward network activations of a frozen transformer using learned element-wise scaling vectors. For each attention layer, IA3 learns a vector l_k that scales key activations and l_v that scales value activations; for feed-forward layers, a vector l_ff scales intermediate activations. This requires only 0.01% of the model's parameters—often fewer than 10,000 total—making it the most parameter-efficient method available. In genomic models, IA3 has proven effective for multi-task learning across diverse downstream assays, though its expressivity ceiling is lower than LoRA for complex prediction tasks.

A memory-optimized extension of LoRA that backpropagates gradients through a frozen, 4-bit quantized pre-trained model into low-rank adapters stored in higher precision (typically BF16). QLoRA uses a novel NormalFloat4 data type that is information-theoretically optimal for normally distributed weights, combined with double quantization to further compress the quantization constants. This enables fine-tuning of massive genomic models like the 2.5B-parameter Nucleotide Transformer on a single consumer GPU with 24GB of VRAM. The technique achieves performance parity with full 16-bit LoRA while reducing memory footprint by 4-8x, democratizing access to large-scale genomic model adaptation.