Parameter-Efficient Fine-Tuning (PEFT)

LoRA freezes the pre-trained weights and injects trainable rank decomposition matrices into the attention layers. Instead of updating the full weight matrix W, it learns a low-rank update ΔW = BA, where B and A are small matrices.

• Parameter reduction: Often reduces trainable parameters by 10,000x compared to full fine-tuning
• No inference latency: The learned matrices can be merged into the frozen weights after training
• Multi-adapter serving: Different LoRA adapters can be swapped efficiently for different tasks or users
• Federated relevance: Only the tiny A and B matrices need to be communicated, not the full model

Adapter modules are small bottleneck neural networks inserted between the layers of a frozen pre-trained model. Each adapter consists of a down-projection to a lower dimension, a non-linearity, and an up-projection back to the original dimension.

• Bottleneck dimension: Typically 1-64, controlling the parameter-accuracy tradeoff
• Placement: Usually inserted after the attention and feed-forward sub-layers in transformers
• Near full accuracy: Achieves performance within 1% of full fine-tuning on many benchmarks
• Sequential processing: Adds a small computational overhead since adapters cannot be fully merged like LoRA

Prefix tuning prepends a sequence of continuous, trainable vectors (soft prompts) to the input or to each transformer layer's key-value cache. The base model remains completely frozen while only these virtual tokens are optimized.

• Virtual tokens: Learned continuous embeddings that steer model behavior without discrete text prompts
• Layer-wise prefix: Prepending to each layer's activations provides deeper conditioning than input-only methods
• Task specialization: A single frozen model can serve dozens of tasks by swapping prefix vectors
• Limitation: Prefix tokens consume sequence length budget, reducing the effective context window for the actual input

Prompt tuning simplifies prefix tuning by appending learnable continuous embeddings only to the input sequence, rather than to every transformer layer. The entire model backbone remains frozen.

• Input-only conditioning: Trainable soft prompts are concatenated with the input embeddings
• Model scale dependency: Performance improves dramatically with model size; highly effective above 10B parameters
• Extreme efficiency: Often requires fewer than 50K trainable parameters total
• Federated advantage: Communicating only a small prompt vector makes on-device personalization extremely bandwidth-efficient

IA³ introduces learnable rescaling vectors that element-wise multiply the keys, values, and feed-forward activations within a frozen transformer. No additional layers or structural modifications are required.

• Rescaling vectors: Learnable vectors l_k, l_v, and l_ff that modulate existing activations
• Parameter count: Adds only 0.01% additional parameters relative to the base model
• Strong few-shot performance: Matches or exceeds full fine-tuning on T0 benchmarks with far fewer parameters
• Mergeable: Like LoRA, the learned rescaling can be absorbed into the frozen weights for zero-inference-overhead deployment

BitFit freezes all weight matrices in the model and trains only the bias terms and task-specific classification heads. This represents the extreme minimal end of the PEFT spectrum.

• Bias vectors only: All linear layer biases and layer normalization biases become trainable
• Parameter count: Typically under 0.1% of total model parameters
• Surprising effectiveness: Achieves competitive results on many GLUE benchmarks despite its simplicity
• Federated use case: Ideal for extremely bandwidth-constrained edge devices where even LoRA's overhead is too high

The dominant PEFT method that freezes pre-trained weights and injects trainable rank-decomposition matrices into attention layers. LoRA dramatically reduces the number of trainable parameters by up to 10,000x compared to full fine-tuning.

• Mechanism: Updates are constrained to a low intrinsic dimension via matrix factorization (A × B).
• Federated Benefit: The tiny adapter size (often < 1MB) minimizes communication overhead and client compute load.
• Merging: At inference, the low-rank matrices can be merged into the frozen weights, adding zero latency.

Small bottleneck neural layers inserted between the existing layers of a frozen transformer. Adapters act as side-pathways that learn task-specific representations while the original backbone remains untouched.

• Architecture: Typically a down-projection, a non-linearity, and an up-projection with a residual connection.
• Isolation: Keeps task-specific knowledge physically separate, preventing catastrophic interference in continual learning.
• Federated Use Case: Clients can train local adapters on sensitive data and share only these lightweight modules with the server.

A PEFT strategy that prepends a sequence of continuous, trainable vectors (a 'soft prompt') to the input or activations at every transformer layer. Prefix tuning keeps the entire language model frozen.

• Virtual Tokens: These learned vectors steer the model's behavior without corresponding to real words.
• Personalization: Ideal for on-device user adaptation where a small set of personal prefix vectors can encode writing style or preferences.
• Communication: Only the compact prefix vectors need to be transmitted during federated aggregation.

A simplified variant of prefix tuning that only prepends trainable soft prompts to the input embedding layer, rather than at every transformer block. Prompt tuning is extremely parameter-efficient but less expressive than full prefix tuning.

• Scalability: Performance on smaller models (< 1B params) can lag, but it becomes highly competitive with model scale.
• Federated Edge: The minuscule payload size makes it ideal for cross-device federated learning on smartphones and IoT sensors.
• Task Switching: Users can swap prompts instantly without reloading model weights.

An ultra-lightweight PEFT method that rescales key, value, and feed-forward activations using learned vectors. IA³ introduces only a handful of new parameters per layer, making it even more compact than LoRA.

• Mechanism: Element-wise multiplication of activations with learned scaling vectors (l_k, l_v, l_ff).
• Efficiency: Achieves strong performance with an order of magnitude fewer parameters than other adapter methods.
• Federated Fit: The extremely low parameter count minimizes both local training time and upstream communication cost.

The server-side process of combining lightweight adapters or soft prompts from multiple clients without accessing raw data. Federated aggregation of PEFT modules is the core mechanism enabling privacy-preserving model personalization.

• FedAvg for Adapters: Standard weighted averaging of LoRA matrices or adapter weights across participating clients.
• Heterogeneity Handling: PEFT's small parameter space makes it more robust to non-IID client data than full-model federated averaging.
• Personalized Layers: Some architectures keep certain adapter layers local (never shared) to preserve user-specific quirks.