Parameter-Efficient Fine-Tuning (PEFT)

Freezes the original pre-trained weights and injects trainable low-rank decomposition matrices into the transformer's attention layers. Instead of updating the full weight matrix W, LoRA learns a change ΔW = BA, where B and A are small, low-rank matrices. This reduces the number of trainable parameters by 10,000x for large models like LLaMA. For clinical applications, multiple lightweight LoRA modules can be trained for distinct tasks—such as radiology summarization and medication extraction—and hot-swapped on a single frozen base model without catastrophic forgetting.

Small bottleneck neural networks inserted between the existing layers of a frozen pre-trained model. Each adapter consists of a down-projection to a smaller dimension, a non-linear activation, and an up-projection back to the original dimension. Only these inserted parameters are trained. In a clinical context, adapters allow a single general-purpose model to be specialized for ICD-10-CM coding, SNOMED CT entity normalization, and PHI de-identification without duplicating the base model, saving gigabytes of storage per task.

Prepends a sequence of continuous, task-specific virtual tokens—a prefix—to the input or to each transformer layer's key and value states. These virtual tokens are learned while the entire model remains frozen. Unlike discrete prompt engineering, prefix tuning optimizes in a continuous embedding space. For medical instruction following, a learned prefix can condition the model to generate responses in the style of a clinical decision support system, ensuring outputs are evidence-based and cautious without modifying the underlying model weights.

A simplified variant of prefix tuning where a small number of trainable tokens are appended only to the input embedding layer, rather than to every transformer block. The model's entire computation graph remains frozen. This method is highly parameter-efficient but was initially less effective on smaller models. With modern large language models, prompt tuning can match full fine-tuning performance on tasks like clinical trial eligibility screening by learning a task-specific embedding that steers the frozen model's attention toward relevant patient attributes.

An ultra-parameter-efficient method that rescales the key, value, and feed-forward network activations using learned element-wise multiplication vectors. No additional weight matrices are introduced. For each attention or feed-forward layer, IA3 learns a single scaling vector lv, lk, or lff that modulates the existing activations. This achieves performance comparable to LoRA while training 10x fewer parameters. In resource-constrained hospital edge deployments, IA3 enables on-device adaptation of a clinical language model to a new physician's dictation style with minimal overhead.

A minimalist PEFT approach that freezes all weight matrices and trains only the bias vectors within the neural network. Despite modifying less than 0.1% of the model's parameters, BitFit can recover a significant portion of full fine-tuning performance on many benchmarks. For a ClinicalBERT model, training only the bias terms on a downstream task like hospital readmission prediction provides a surprisingly strong baseline, making it a useful diagnostic tool to assess task difficulty before applying more complex PEFT methods.

The most widely adopted PEFT method. LoRA freezes the original pre-trained weights and injects trainable low-rank decomposition matrices into the attention layers of a transformer. Instead of updating a full weight matrix W, it learns a low-rank update ΔW = BA, where B and A are small, thin matrices.

• Storage: A full 175B model fine-tune might be 350GB; a LoRA adapter is often < 10MB.
• Latency: No additional inference overhead, as the learned matrices can be merged with frozen weights.
• Clinical Use: Allows a single base model (e.g., LLaMA) to be quickly adapted to radiology, cardiology, or billing codes by swapping lightweight adapters.

A precursor to LoRA that inserts small, trainable bottleneck neural layers between the existing layers of a frozen transformer. Each adapter module projects the hidden state down to a smaller dimension, applies a non-linearity, and projects it back up.

• Architecture: Typically a down-projection, a GeLU activation, and an up-projection.
• Trade-off: Introduces a small, measurable latency increase at inference time, unlike LoRA.
• Legacy: While largely superseded by LoRA for LLMs, the concept remains foundational for multi-task learning in computer vision and older BERT-based clinical models.

A PEFT technique that prepends a small set of trainable continuous vector embeddings—a 'soft prompt'—to the input sequence. The entire base model remains frozen; only these virtual tokens are learned.

• Mechanism: Instead of hand-crafting a discrete text prompt, gradient descent optimizes a sequence of continuous vectors in the embedding space.
• Parameter Count: Only a few tens of thousands of parameters per task.
• Clinical Application: Effective for conditioning a general model to adopt a specific clinical persona or output style (e.g., 'respond as a cardiologist') without modifying any model weights.

A precursor to prompt tuning that learns a small continuous vector prepended to the keys and values at every transformer layer, not just the input embedding. This provides a stronger conditioning signal than prompt tuning.

• Scope: Injects trainable parameters into the multi-head attention blocks of all layers.
• Performance: Often matches full fine-tuning on generative tasks like summarization.
• Clinical Relevance: Highly effective for adapting a frozen model to generate structured clinical summaries or discharge instructions, where deep conditioning on the task format is critical.

An extension of LoRA that backpropagates gradients through a frozen, 4-bit quantized base model into a LoRA adapter. This reduces the memory footprint of the base model by up to 4x, enabling the fine-tuning of a 65B parameter model on a single 48GB GPU.

• Innovation: Uses a 4-bit NormalFloat data type and double quantization to preserve accuracy.
• Paging: Employs unified memory paging to handle gradient checkpointing spikes.
• Impact: Democratizes clinical NLP by allowing researchers to fine-tune massive models on sensitive, on-premise hospital hardware without requiring expensive multi-GPU clusters.

The primary risk that PEFT methods are designed to mitigate. When a model is fully fine-tuned on a narrow domain like radiology reports, it can abruptly overwrite the general language understanding and reasoning capabilities learned during pre-training.

• Mechanism: Gradient updates aggressively shift weights to minimize loss on the new, narrow data distribution.
• PEFT Solution: By freezing the vast majority of weights, LoRA and adapters preserve the original knowledge manifold while learning a task-specific delta.
• Clinical Risk: A model that forgets general anatomy while learning chest X-rays is clinically dangerous.