AdapterFusion

AdapterFusion operates in two distinct, sequential phases to separate concerns and prevent interference.

• Stage 1: Knowledge Extraction: Multiple standard task adapters are trained independently on their respective datasets. The base model remains frozen. Each adapter becomes a compact, expert repository for its specific task.
• Stage 2: Knowledge Composition: A new, secondary fusion layer is introduced. This layer is trained (with the base model and all adapters frozen) to learn an attention-based weighting over the outputs of the pre-trained adapters for a new target task or dataset.

This separation prevents negative transfer during initial adapter training and allows the fusion mechanism to focus purely on optimal combination strategies.

The core innovation is a trainable attention mechanism that dynamically combines adapter outputs. This layer is a small neural network added on top of the transformer stack.

• Mechanism: For a given input, the fusion layer computes a set of attention scores (one per pre-trained adapter). These scores determine how much to weigh the hidden state contributions from each adapter's output.
• Dynamic Weighting: The attention scores are computed per input token, allowing the model to attend to different adapters for different parts of the sequence or for different types of queries.
• Parameter Efficiency: The fusion layer adds only a minimal number of new parameters (e.g., a query vector and attention weights), maintaining the method's overall efficiency.

AdapterFusion enables positive knowledge transfer across tasks while inherently avoiding catastrophic forgetting.

• Preserved Expertise: Because the pre-trained adapters are frozen during the fusion stage, their task-specific knowledge is perfectly preserved. There is no risk of overwriting or degrading performance on their original tasks.
• Selective Transfer: The fusion layer learns to extract and combine only the relevant knowledge from each adapter for the new objective. It can ignore adapters that provide irrelevant or conflicting signals.
• Contrast with Multi-Task Joint Training: Unlike joint training where a single model must balance all tasks simultaneously—often leading to compromises—AdapterFusion builds on solidified, high-performance experts.

The architecture treats adapters as modular, composable units, enabling flexible system design.

• Adapter as a Module: Each adapter is a self-contained, plug-and-play component. A library of adapters can be maintained for various skills (e.g., sentiment analysis, named entity recognition, summarization).
• Incremental Learning: New tasks can be integrated by simply training a new adapter (Stage 1) and then updating the fusion layer (Stage 2) to incorporate it, without retraining existing components.
• Cross-Lingual and Cross-Domain: Adapters can be trained on different languages or domains. The fusion layer can then learn to compose a multilingual or cross-domain model by blending the appropriate experts.

The method maintains the core efficiency benefits of adapter-based tuning while adding multi-task capabilities.

• Frozen Base Model: The large pre-trained transformer weights are never updated, saving significant GPU memory and compute.
• Minimal Trainable Parameters: Only the small adapter parameters (in Stage 1) and the even smaller fusion layer parameters (in Stage 2) are trained. Total trainable parameters are typically <1% of the base model.
• Efficient Inference: During inference, only the relevant adapters (as weighted by the fusion layer) need to be activated, avoiding the computational cost of running a separate model per task.

AdapterFusion is distinct from simpler adapter composition techniques.

• vs. AdapterStacking: Stacking places adapters sequentially within layers, forcing a fixed, serial flow of information. AdapterFusion uses parallel adaptation, where all adapters process the same base model output, and the fusion layer combines them. This allows for more flexible, non-linear interactions.
• vs. Single Adapter Multi-Task Training: Training one shared adapter on a mixture of multiple tasks often leads to performance degradation due to interference. AdapterFusion's dedicated adapters avoid this.
• vs. Model Merging (Task Arithmetic): While model merging combines weight deltas arithmetically, AdapterFusion performs a learned, input-dependent combination of adapter outputs, which is often more precise and robust.

An adapter is a small, trainable neural network module (typically a two-layer feed-forward network with a bottleneck) inserted into the layers of a frozen pre-trained model. During fine-tuning, only the adapter's parameters are updated, enabling efficient task-specific adaptation. Adapters are the fundamental building blocks used in the first stage of AdapterFusion.

• Architecture: Usually placed after the feed-forward network or attention sub-layer in a transformer.
• Key Property: Introduces a parameter-efficient bottleneck, often reducing trainable parameters to <1% of the original model.

Multi-task learning (MTL) is a machine learning paradigm where a single model is trained to perform multiple related tasks simultaneously, sharing representations to improve generalization and data efficiency. AdapterFusion is a PEFT strategy for MTL, where knowledge from multiple task-specific adapters is combined.

• Contrast with AdapterFusion: Standard MTL often trains all tasks jointly, risking negative interference. AdapterFusion uses a two-stage process (train adapters in isolation, then fuse) to mitigate this.
• Goal: Leverage shared and task-specific knowledge to improve performance on all target tasks.

Mixture-of-Experts (MoE) is a neural network architecture that consists of multiple expert networks and a gating network that routes each input to a subset of relevant experts. AdapterFusion's attention-based fusion layer is conceptually similar, acting as a router that dynamically combines the outputs of multiple adapter experts.

• Key Difference: In classic MoE, experts are trained jointly. In AdapterFusion, the 'experts' (adapters) are pre-trained independently, and the gating mechanism (fusion layer) is learned afterward to compose them.
• Shared Principle: Both enable conditional computation and capacity scaling without a proportional increase in inference cost for a given input.

Model merging is a technique for combining the parameters of multiple neural networks into a single unified model, often to achieve multi-task capabilities. AdapterFusion is a form of model merging at the adapter level.

• Task Arithmetic: A related merging technique where 'task vectors' (the difference between fine-tuned and base model weights) are added together. AdapterFusion merges adapter outputs, not base weights.
• Advantage of AdapterFusion: By merging via a learned fusion layer rather than simple weight averaging, it can learn more sophisticated, input-dependent combinations of adapter knowledge.

Parameter-efficient fine-tuning (PEFT) encompasses all methods that adapt a large pre-trained model to downstream tasks by updating only a small fraction of its total parameters. AdapterFusion is a PEFT method designed for the multi-task setting.

• Core PEFT Methods: Includes Adapters, LoRA, Prefix Tuning, and Prompt Tuning.
• AdapterFusion's Place: It operates on top of adapter-based PEFT, providing a strategy to compose multiple PEFT modules (adapters) efficiently.
• Primary Benefit: Dramatically reduces storage and memory requirements compared to full fine-tuning, as only small adapter and fusion weights are saved per task.

The attention mechanism is a core component of transformer architectures that allows a model to weigh the importance of different parts of the input sequence. AdapterFusion employs a dedicated attention-based layer to perform its fusion.

• Role in AdapterFusion: The fusion layer uses attention to compute a weighted combination of the hidden states produced by all task adapters for a given input.
• Process: For each input token, the fusion attention scores determine how much to 'attend to' the output of each task-specific adapter, enabling dynamic, context-aware knowledge composition.