Parameter-Efficient Fine-Tuning (PEFT)

LoRA freezes the pretrained weights of a genomic model and injects trainable low-rank decomposition matrices into the attention layers. For a weight matrix W, the update is constrained to ΔW = BA, where B and A are small rank-r matrices.

• Parameter Reduction: A model with 100M parameters may require updating only 0.1-1% of weights.
• Genomic Application: Fine-tune DNABERT for promoter prediction on a single GPU.
• Inference Efficiency: Learned adapters can be merged back into frozen weights, adding zero latency overhead.
• Multi-Task Serving: A shared base model can serve dozens of downstream tasks by swapping lightweight LoRA modules.

Adapter modules are small bottleneck neural networks inserted between the existing layers of a frozen genomic foundation model. Each adapter consists of a down-projection to a lower dimension, a non-linearity, and an up-projection back to the original hidden size.

• Serial Architecture: Adapters are placed sequentially after attention and feed-forward sub-layers.
• Task Isolation: Each downstream genomic task gets its own adapter, preventing catastrophic interference.
• Parameter Budget: Typically adds 1-5% additional parameters per task.
• Use Case: Adapting Enformer for tissue-specific gene expression prediction without retraining the entire 200M+ parameter model.

Prefix tuning prepends a sequence of learnable continuous vectors—virtual tokens—to the input of each Transformer layer. These soft prompts steer the frozen model's attention mechanism toward task-specific behavior without modifying any pretrained weights.

• Mechanism: The prefix vectors act as synthetic context that conditions the model's activations.
• Genomic Context: A prefix can encode task instructions like "predict splice donor site" or "classify enhancer activity."
• Parameter Count: Only the prefix vectors are trained, often totaling <0.1% of the original model size.
• Limitation: Less expressive than LoRA for tasks requiring complex structural understanding of regulatory grammar.

Prompt tuning is a simplified variant of prefix tuning where learnable embeddings are added only to the input layer, rather than at every Transformer block. This makes it extremely parameter-efficient but less expressive for complex genomic tasks.

• Input-Only: Soft tokens are concatenated to the k-mer embeddings at the first layer.
• Genomic Example: Tuning a DNA language model for zero-shot variant effect scoring by learning a task-specific input prompt.
• Scalability: A single prompt vector may contain only a few thousand parameters.
• Best For: Simple classification tasks where deep layer-wise conditioning is unnecessary.

IA3 introduces learnable rescaling vectors that element-wise multiply the keys, values, and feed-forward activations of a frozen Transformer. This provides a hyper-minimal adaptation mechanism with no additional matrix multiplications.

• Mechanism: Learned vectors l_k, l_v, and l_ff scale existing activations.
• Parameter Count: Adds only 0.01% additional parameters—often just a few thousand floats.
• Genomic Suitability: Effective for adapting large genomic models like HyenaDNA for sequence classification when compute budgets are severely constrained.
• Comparison: Matches or exceeds LoRA performance on many NLP benchmarks; emerging in computational biology.

BitFit freezes all weight matrices in the genomic model and trains only the bias terms. This surprisingly simple approach can recover a significant fraction of full fine-tuning performance on many tasks.

• Mechanism: Updates are restricted to the additive bias vectors in linear layers, layer norms, and attention projections.
• Parameter Reduction: Bias terms constitute <0.1% of total parameters in typical Transformer architectures.
• Genomic Use: A baseline for evaluating whether a downstream task requires complex adaptation or can be solved with minimal parameter changes.
• Limitation: Underperforms LoRA and adapters on tasks requiring substantial representational shifts, such as cross-species transfer.

The most widely adopted PEFT method that freezes the pretrained model weights and injects trainable rank-decomposition matrices into the attention layers. For a weight matrix W, LoRA learns a low-rank update ΔW = BA, where B and A are small matrices.

• Rank (r): A hyperparameter (often 8-64) controlling the number of trainable parameters
• Alpha (α): Scales the LoRA update relative to the original weights
• Target Modules: Typically applied to query (Q) and value (V) projection matrices in self-attention
• For a genomic model like DNABERT, LoRA can reduce trainable parameters from 110M to under 500K while retaining >95% of full fine-tuning performance on promoter prediction

Small bottleneck neural networks inserted between the existing layers of a frozen pretrained model. Each adapter consists of a down-projection to a lower dimension, a non-linearity, and an up-projection back to the original dimension, with a residual skip connection.

• Bottleneck dimension: Controls the parameter-accuracy trade-off
• Serial vs. Parallel: Adapters can be placed sequentially after attention/FFN layers or in parallel to them
• Pfeiffer vs. Houlsby: Two common architectural variants differing in placement and residual connection design
• Particularly effective for multi-task genomic settings where separate adapters can be trained for variant effect prediction, splice site detection, and promoter identification without interference

A PEFT approach that prepends a sequence of continuous, learnable vectors (a prefix) to the input of each Transformer layer, steering the model's behavior without modifying any pretrained weights. The prefix acts as virtual tokens that condition the model for a specific downstream task.

• Prefix length: Typically 10-200 virtual tokens, controlling the capacity of the adaptation
• Reparameterization: The prefix is often generated by a small MLP to stabilize training
• Only the prefix vectors and the reparameterization network are updated during training
• In genomic applications, prefix tuning can condition a DNA language model to attend to promoter regions or splice junctions by learning task-specific attention biases

A simplified variant of prefix tuning where learnable continuous embeddings are prepended only to the input layer rather than all Transformer layers. The model learns a soft prompt that adapts its behavior for a specific task while keeping the entire architecture frozen.

• Soft prompts: Continuous vectors in embedding space, not human-readable tokens
• Initialization: Prompts are often initialized from task-relevant vocabulary embeddings (e.g., 'promoter', 'enhancer' for genomics)
• Length: Typically 5-100 tokens, with longer prompts providing more adaptation capacity
• Scales efficiently to thousands of tasks since each task requires storing only a small prompt vector rather than a full model copy

An ultra-lightweight PEFT method that introduces learnable rescaling vectors to modulate key, value, and feed-forward network activations via element-wise multiplication. IA3 learns vectors l_k, l_v, and l_ff that scale the respective activations.

• Mechanism: (l_k ⊙ K), (l_v ⊙ V), and (l_ff ⊙ FFN(x)) where ⊙ is element-wise multiplication
• Parameter count: Adds only (d_k + d_v + d_ff) parameters per attention layer
• Performance: Matches or exceeds LoRA on many benchmarks while using 10x fewer parameters
• For genomic models processing long sequences, IA3's minimal memory footprint enables efficient adaptation on consumer GPUs

The phenomenon where a model loses previously acquired knowledge when adapted to a new task. PEFT methods inherently mitigate this by keeping the pretrained backbone frozen, but the adapter parameters themselves can still overfit to the new task distribution.

• Regularization: Weight decay and early stopping prevent adapter over-specialization
• Elastic Weight Consolidation (EWC): Penalizes changes to parameters important for previous tasks
• Adapter Fusion: Combines multiple task-specific adapters via attention-weighted mixing for multi-task inference
• In genomic contexts, a model fine-tuned on human promoters may lose sensitivity to conserved motifs in other species without proper regularization