AdapterFusion

AdapterFusion operates in two distinct, sequential phases to separate knowledge acquisition from knowledge composition.

• Stage 1: Knowledge Extraction: Multiple standard adapter layers are trained independently on different tasks. The base model remains frozen, and each adapter learns a compact, task-specific representation.
• Stage 2: Knowledge Composition: A new fusion layer is introduced. This layer is trained on the target task while the base model and all pre-trained adapters remain frozen. It learns to dynamically combine the outputs of the frozen adapters.

This decoupling prevents negative transfer and catastrophic forgetting during the fusion stage, as the source adapters' knowledge is fixed.

The core innovation is a trainable fusion mechanism that learns to weight and combine adapter outputs contextually.

• Architecture: The fusion layer is typically a multi-head attention block. The frozen adapter outputs serve as the 'values' and 'keys', while a learned query (often derived from the transformer's hidden state) attends over them.
• Dynamic Weighting: For each input token, the attention mechanism computes a unique weighted combination of the available adapter outputs. This allows the model to selectively attend to different sources of knowledge based on the current context.
• Contrast with Averaging: This is superior to simple averaging or concatenation, as it enables nuanced, input-dependent composition of expertise.

The method achieves multi-task capability with minimal parameter growth, leveraging pre-trained modular components.

• Efficiency: Only the parameters of the small fusion layer are trained in Stage 2. The base model (billions of parameters) and the pre-trained adapters (a few million each) are entirely frozen. This makes AdapterFusion far more efficient than full fine-tuning for each new task combination.
• Composability: Once a library of task-specific adapters is built (e.g., for sentiment analysis, named entity recognition, natural language inference), new composite tasks can be addressed by simply training a new fusion layer to combine the relevant existing adapters. This enables modular reuse of knowledge.

A primary goal is to leverage multiple knowledge sources without the performance degradation common in multi-task learning.

• Problem: Jointly training a single model on multiple tasks often leads to negative transfer, where learning one task harms performance on another due to conflicting gradient signals.
• Solution: By first training adapters in isolation (Stage 1), each one becomes a pure, uncontaminated expert. The fusion layer (Stage 2) then learns a composition function without altering these expert representations. This architecture inherently isolates task-specific parameters, preventing destructive interference during the fusion training process.

AdapterFusion sits within the broader delta tuning family but is distinct in its focus on composition.

• vs. Single Adapters/LoRA: Standard adapter layers or LoRA adapt a model to one task. AdapterFusion uses these as building blocks for multi-task learning.
• vs. Prompt Tuning: Methods like prefix tuning or prompt tuning condition a frozen model with learned vectors. AdapterFusion conditions the model with the outputs of multiple frozen, task-conditioned modules.
• vs. Mixture-of-Experts (MoE): Both use routing mechanisms. However, sparse MoE routes tokens to different parameter blocks within a single model. AdapterFusion routes context to the outputs of different complete task experts (the adapters).

This technique is powerful for specific scenarios but has inherent constraints.

• Ideal Use Cases:
  • Building a unified model for a closely-related family of tasks (e.g., multiple text classification tasks in customer support).
  • Continual learning settings where new tasks arrive sequentially, and old task performance must be preserved.
  • Scenarios with strict parameter budgets for deployment but a need for multi-task capability.
• Key Limitations:
  • Sequential Bottleneck: Requires pre-training a high-quality adapter for each source task, which can be time-consuming.
  • Static Adapter Library: The fused model cannot incorporate knowledge from a new task without adding a new pre-trained adapter and retraining the fusion layer.
  • Increased Latency: While parameter-efficient, the forward pass requires computing the output of all relevant adapters before fusion, adding computational overhead compared to a single-adapter model.

The core problem AdapterFusion is designed to solve. Multi-task learning is a paradigm where a single model is trained to perform multiple distinct tasks simultaneously, with the goal of improving generalization and data efficiency through shared representations. AdapterFusion's two-stage approach—training independent adapters then fusing them—is a specific strategy for positive knowledge transfer without negative interference (catastrophic forgetting).

A related architectural paradigm for conditional computation. A Mixture-of-Experts model consists of multiple sub-networks (experts) and a gating network that dynamically routes each input to a sparse combination of these experts. While both MoE and AdapterFusion leverage multiple specialized components, they differ fundamentally:

• MoE: Experts are part of the base model architecture; routing is per-token.
• AdapterFusion: Adapters are task-specific add-ons; fusion is a learned, static combination applied during inference for a given task.

The overarching category for methods like AdapterFusion. Delta tuning refers to any fine-tuning technique that updates only a small subset of a model's parameters (the 'delta' or change) while keeping the vast majority of pre-trained weights frozen. This family includes:

• Adapter-based methods (Adapters, AdapterFusion)
• Prompt-based methods (Prompt Tuning, Prefix Tuning)
• Low-rank methods (LoRA) The goal is to achieve performance comparable to full fine-tuning with a drastically reduced number of trainable parameters.

A conceptual parallel to the adapter weights learned in the first stage of AdapterFusion. A task vector is defined as the arithmetic difference between the weights of a model fine-tuned on a specific task and the weights of the original pre-trained model (θ_task - θ_pretrained). This vector encapsulates the directional change needed for task adaptation. While adapters are additive modules, task vectors represent the change in the core parameters themselves. Both concepts capture task-specific knowledge separate from the base model.

A post-training technique with similar goals to the fusion stage. Model merging involves combining the parameters of multiple models (e.g., models fine-tuned on different tasks) into a single model, often through simple arithmetic operations like weighted averaging (e.g., Task Arithmetic). AdapterFusion can be seen as a form of structured, learned merging, where the fusion layer learns how to best combine the outputs of fixed, task-specific adapter modules, rather than merging their underlying parameters directly.