Adapter

The canonical adapter module follows a bottleneck design to enforce parameter efficiency. It projects the input activation (dimension d) down to a smaller bottleneck dimension r (via a down-projection matrix), applies a non-linearity, then projects back up to dimension d (via an up-projection matrix). This creates a severe parameter reduction, often with r << d (e.g., d=768, r=48). The original transformer layer remains frozen, with the adapter learning a task-specific residual function: output = Layer(x) + Adapter(Layer(x)).

Adapters are inserted at specific injection points within the transformer block. The two most common placements are:

• Post-Attention: After the multi-head attention module and its residual connection.
• Post-Feed-Forward: After the feed-forward network (FFN) and its residual connection. Placing adapters after these core sub-layers allows them to transform the intermediate representations most relevant for task-specific reasoning. Some architectures, like AdapterFusion, use adapters at both points. The choice of injection point is a key hyperparameter affecting adaptation quality and interference with the pre-trained knowledge.

Despite training only ~0.5-8% of a model's parameters, well-tuned adapters can achieve performance comparable to full fine-tuning on many NLP and vision tasks. This efficiency stems from the adapter's role as a learned residual function. The frozen backbone provides robust, general-purpose features, while the adapter makes small, targeted adjustments. For example, on the GLUE benchmark, adapter-tuning of BERT-large often matches within 1-2% of the full fine-tuning accuracy, while being drastically more parameter- and memory-efficient.

Adapters enable elegant compositionality. Multiple task-specific adapters can be trained independently on a single frozen backbone. For inference, the correct adapter is swapped in dynamically, allowing one model to serve numerous tasks. Advanced methods like AdapterFusion learn to combine multiple pre-trained adapters for a new task. Furthermore, adapters can be stacked or interleaved with other PEFT methods (e.g., prefix tuning) within frameworks like UniPELT, where a gating mechanism learns to activate the most effective technique per layer.

The adapter paradigm extends beyond language to vision, audio, and multimodal models. Specialized variants include:

• Visual Adapters (ViT Adapters): Integrated into Vision Transformers for tasks like segmentation.
• VL-Adapters: Inserted into vision-language models (e.g., CLIP, BLIP) to adapt cross-modal alignment for VQA or captioning.
• Audio Adapters: Used with pre-trained audio models like Wav2Vec2 for efficient speech task adaptation.
• Cross-Modal Adapters: Specifically designed to fine-tune the fusion mechanisms between modalities in a frozen multimodal backbone.

A key challenge with serial adapter insertion is added latency at inference. The AdapterDrop technique addresses this by selectively removing (dropping) adapters from lower transformer layers during training and inference. Since higher layers are more task-specific, dropping lower-layer adapters results in significant speedups (e.g., 20-60%) with only a minor drop in task performance. This makes adapter-based models more viable for production deployments where latency is critical.

Adapters are extensively used to adapt large language models to specialized domains like biomedicine, legal, or finance. A frozen BERT or RoBERTa model can be equipped with domain-specific adapters, allowing it to understand niche terminology and context with minimal compute.

• Example: Training an adapter on PubMed abstracts to improve performance on biomedical named entity recognition (NER).
• Key Benefit: Maintains the model's general linguistic knowledge while efficiently acquiring domain expertise.

Instead of training separate full models, multiple task-specific adapters can be attached to a single frozen backbone. This creates a highly parameter-efficient multi-task system.

• Architecture: A shared frozen backbone (e.g., T5) hosts distinct adapters for translation, summarization, and question-answering.
• Advanced Technique: AdapterFusion can be applied as a second stage to learn how to combine knowledge from these pre-trained adapters for a new, composite task.

Adapters are inserted into Vision Transformers (ViTs) and multimodal models to adapt them for downstream tasks without full fine-tuning.

• Visual Adapters: Used for adapting a pre-trained ViT to image segmentation or object detection.
• VL-Adapters: Lightweight modules in models like CLIP or BLIP enable efficient adaptation to vision-language tasks such as visual question answering (VQA) or domain-specific image retrieval.
• Cross-Modal Adapters: Facilitate efficient tuning of interaction layers between modalities in frozen multimodal architectures.

Adapters mitigate catastrophic forgetting in continual learning scenarios. When a model must learn tasks A, B, then C sequentially, a new adapter is trained for each task while previous adapters remain frozen.

• Process: The backbone is always frozen. Task A's adapter is trained and saved. For Task B, a new adapter is added and trained, leaving Adapter A untouched.
• Result: The model retains performance on all learned tasks without degrading on earlier ones, as only the relevant adapter is activated per task.

The small size of adapter weights (often <1% of base model) makes them ideal for edge AI applications. Only the tiny adapter file needs to be updated or swapped on a device, not the massive base model.

• Use Case: A smartphone app using a large, frozen on-device vision model. Different adapters can be downloaded to enable new features (e.g., pet breed identification, plant disease detection).
• Advantage: Drastically reduces the bandwidth and storage overhead for model updates compared to full model replacements.

Pre-trained audio models like Wav2Vec2 or HuBERT are adapted using audio-specific adapters for tasks such as automatic speech recognition (ASR) for new accents or audio event classification.

• Implementation: Adapters are inserted into the transformer layers of the frozen audio encoder.
• Efficiency: Allows rapid customization for specific acoustic environments or languages while preserving the model's general speech representation capabilities learned during pre-training.

Low-Rank Adaptation (LoRA) is a dominant PEFT method that, like adapters, avoids updating the full model. Instead of inserting new modules, LoRA injects trainable low-rank matrices in parallel with existing weight matrices (e.g., in attention layers). The update is expressed as W + BA, where B and A are low-rank. This approach modifies the forward pass directly without adding inference latency, a key difference from serial adapters.

• Core Mechanism: Approximates the weight delta (ΔW) with a low-rank decomposition.
• Key Advantage: No additional latency during inference as the adapted weights can be merged back into the base model.
• Common Use: The go-to method for fine-tuning large language models (LLMs) due to its balance of efficiency and performance.

Prefix Tuning and Prompt Tuning are PEFT methods that condition a frozen model by optimizing continuous embeddings prepended to the input or hidden states.

• Prefix Tuning: Inserts trainable vectors into the key and value sequences of the transformer's attention mechanism at every layer.
• Prompt Tuning: Optimizes a small set of continuous token embeddings (soft prompts) only at the input layer.
• Contrast with Adapters: These methods act as a form of contextual conditioning rather than transforming intermediate activations via a feed-forward network. They typically require fewer parameters than adapters but can be less effective on encoder-only models or smaller-scale tasks.

The frozen backbone is the large, pre-trained base model (e.g., BERT, GPT, ViT) whose parameters are kept entirely static during adapter-based fine-tuning. This is the foundational principle enabling parameter efficiency.

• Purpose: Preserves the general knowledge and linguistic/visual representations learned during massive pre-training.
• Benefit: Dramatically reduces memory footprint (no gradients for most weights), prevents catastrophic forgetting, and allows rapid adaptation to new tasks by only training the small adapter modules inserted into this frozen structure.

AdapterFusion is a two-stage, knowledge-composition technique built on top of standard adapters.

• Stage 1: Multiple task-specific adapters are trained independently on different datasets.
• Stage 2: A new composition layer is trained to learn how to dynamically combine (or "fuse") the outputs of these frozen, expert adapters for a new, target task.
• Significance: It enables transfer learning across tasks without forgetting, moving beyond single-task adaptation. The base model and pre-trained adapters remain frozen, with only the lightweight fusion layer being trained.

Injection points refer to the specific architectural locations within a neural network where parameter-efficient modules like adapters are inserted. Strategic placement is critical for performance.

• Common Locations in Transformers:
  • After the multi-head attention sub-layer (MHSA).
  • After the feed-forward network (FFN) sub-layer.
  • Both positions (sequential adapters).
• Design Impact: The choice affects how task-specific signals flow through the network. Adapters placed after the FFN are often considered most impactful, as they modify the transformed features before the residual connection.

Delta weights (Δ) and task vectors are conceptual frameworks for understanding the output of PEFT methods like adapters and LoRA.

• Delta Weights: The small set of learned parameter changes applied to the frozen model. For adapters, this is the entire function f(adapter(x)) applied to activations.
• Task Vector: Defined as the arithmetic difference θ_task - θ_base between a fully fine-tuned model and the base model. In PEFT, the adapter's learned weights approximate this task vector in a highly compressed form.
• Application: These concepts enable model merging, where deltas from multiple tasks can be algebraically combined (e.g., added) to create a multi-task model.