Adapter Composition

Sequential stacking involves chaining adapter modules in a defined order within the model's architecture. This method is used for multi-step tasks where the output of one adapted function serves as the input to the next.

• Example: A model could use a domain adapter for legal text, followed by a task adapter for contract summarization.
• Mechanism: Adapters are typically inserted in series after the feed-forward network in successive transformer blocks.
• Consideration: The order of adapters can be critical and may need to be learned or predetermined based on task dependencies.

Parallel fusion, exemplified by AdapterFusion, combines knowledge from multiple pre-trained, task-specific adapters in parallel. Instead of stacking, it learns to compute a weighted combination of their outputs.

• Key Innovation: The base model and source adapters remain frozen. A new composition layer is trained to learn attention-like weights over the outputs of the source adapters.
• Use Case: Efficiently performs a new task (e.g., sentiment analysis on medical notes) by leveraging knowledge from existing adapters (e.g., a medical domain adapter and a general sentiment adapter).
• Benefit: Prevents catastrophic interference and enables knowledge transfer without retraining source adapters.

Dynamic routing employs a Mixture-of-Adapters (MoA) architecture, where a gating network selects or blends multiple expert adapters for each input.

• Mechanism: A lightweight router network (e.g., a linear layer) computes scores for each adapter in a shared pool, activating only the top-k.
• Advantage: Enables conditional computation, where different data samples activate different specialized adapters, improving efficiency and specialization.
• Example: In a multilingual customer service model, the router could activate a French-language adapter for French queries and a technical-support task adapter for troubleshooting inputs, combining them as needed.

Static weight merging is a post-training method that combines the parameters of multiple trained adapters into a single, unified adapter module through arithmetic operations.

• Common Techniques: Simple element-wise averaging or task arithmetic (adding weighted adapter deltas).
• Process: Individual adapters (e.g., for tasks A, B, C) are trained separately. Their weights are then merged, creating a multi-task adapter.
• Benefit & Limitation: Eliminates runtime overhead from multiple modules but can lead to performance trade-offs and interference compared to dynamic methods. Useful for creating a consolidated model for deployment.

Adapter composition is a core technique for continual learning, allowing a model to learn new tasks sequentially without forgetting previous ones.

• Strategy: Train a new task-specific adapter for each incoming task while keeping the base model and prior adapters frozen.
• Knowledge Recall: At inference, the correct adapter is selected based on task ID, or a routing mechanism chooses the relevant adapter.
• Advantage: Provides a near-perfect solution to catastrophic forgetting, as the parameters for old tasks are preserved intact in their respective adapters. The model's total capacity grows linearly with the number of tasks via added adapters.

Hierarchical composition structures adapters in a tree-like fashion to model complex, nested task or domain relationships.

• Architecture: Higher-level adapters (e.g., for a broad domain like biology) feed into lower-level, more specialized adapters (e.g., for genomics or proteomics).
• Flow: The activation of a specialized adapter is gated by or receives input from its parent domain adapter.
• Use Case: Effective for enterprise settings with structured knowledge hierarchies, such as a financial model with adapters for finance -> equity-research -> biotech-sector. This mirrors organizational data ontologies and enables efficient knowledge sharing.

Adapter composition allows a single model to perform multiple tasks by dynamically combining relevant adapters. For instance, a model can use a sentiment analysis adapter and a named entity recognition adapter simultaneously to analyze a product review for both emotion and key entities.

• AdapterFusion learns to combine outputs from task-specific adapters (e.g., for QA, summarization) to tackle new, unseen tasks.
• Adapter Merging (e.g., weight averaging) creates a unified adapter from several specialized ones, enabling a single forward pass for multi-task inference.

Compose adapters trained on different data domains or languages to handle inputs that span multiple contexts. This is critical for enterprise applications dealing with varied data sources.

• A customer support model can combine a legal domain adapter (for terms of service) with a technical support domain adapter to answer complex queries.
• For global applications, a language adapter for Spanish and a finance domain adapter can be composed to process Spanish financial documents without a dedicated Spanish-finance model.

Adapter composition integrates information from different modalities (text, image, audio) by fusing specialized adapters within a foundational model. This is more efficient than training a full multi-modal model from scratch.

• A vision-language model can use a pre-trained visual question answering adapter composed with a domain-specific text adapter (e.g., for medical reports) to answer detailed questions about radiology images.
• Multimodal adapters for audio and text can be composed to enable complex tasks like generating meeting minutes from an audio transcript and associated slides.

As new tasks or data distributions emerge, adapter composition enables incremental learning without catastrophic forgetting. New adapters are trained for new tasks and composed with existing ones.

• A model deployed in production learns a new task (e.g., code generation) via a new adapter, which can be used standalone or composed with existing documentation summarization adapters.
• This approach maintains a library of adapters, allowing the system to evolve over time by selectively activating different compositions.

Advanced composition uses a routing network to dynamically select the most relevant adapters for each input, creating an efficient, input-specific model architecture.

• In a Mixture-of-Adapters (MoA) system, a gating network evaluates the input and activates a sparse combination of expert adapters (e.g., for legal, medical, or financial language).
• This reduces compute compared to always using all adapters, as only a small subset of parameters are engaged per token, mimicking the efficiency of large MoE models.

While adapters add overhead, smart composition strategies like AdapterDrop can prune less critical adapter layers during inference to regain speed.

• For latency-sensitive applications (e.g., real-time chat), adapters in early or late transformer blocks can be dynamically skipped based on the input complexity.
• Adapter pruning and quantization can be applied to the composed adapter set itself, further compressing the efficient footprint for edge deployment.

AdapterFusion is a two-stage knowledge composition technique. First, multiple task-specific adapters are trained independently. Second, a new fusion layer is trained to learn a weighted combination of the frozen adapters' outputs for a new, composite task. This allows the model to leverage knowledge from multiple source tasks without catastrophic forgetting.

• Stage 1: Train isolated adapters for tasks A, B, C.
• Stage 2: Freeze adapters, train a fusion layer to combine them for task D.
• Benefit: Enables cross-task knowledge transfer while preventing negative interference.

Mixture-of-Adapters (MoA) is a dynamic composition architecture inspired by mixture-of-experts. Multiple adapter modules are made available within a model block, and a learned router network selects which subset of adapters to activate for a given input.

• Dynamic Routing: The router computes gating scores based on the input, sparsely activating relevant adapters.
• Specialization: Different adapters can specialize in different linguistic features or domains (e.g., syntax, semantics, medical jargon).
• Efficiency: Sparse activation keeps computational cost low while providing a large capacity of adaptable knowledge.

Adapter Merging is a static, post-training method to combine multiple adapters into a single module, typically through simple arithmetic operations on their parameters. This creates a unified adapter for multi-task inference without runtime routing overhead.

• Common Techniques: Weight averaging (mean or task-arithmetic) or linear combinations of adapter weights.
• Use Case: Deploying a single model that performs well on several tasks (e.g., sentiment, toxicity, and topic classification) without switching modules.
• Advantage: Eliminates the need for a separate fusion layer or router, simplifying the inference graph.

Adapter Routing is the control mechanism that determines adapter activation in a compositional system like Mixture-of-Adapters. It is typically a small, trainable neural network (e.g., a linear layer followed by a softmax or top-k selection) that computes gating values from the input or hidden states.

• Function: Maps input context to a distribution over available adapters.
• Sparsity: Often encourages sparse gating (e.g., top-2 selection) to improve efficiency and specialization.
• Learning: The router is trained concurrently with the selected adapters, allowing the system to learn which adapters are useful for which input patterns.

A Task-Specific Adapter is a fundamental building block for composition. It is a small neural module (e.g., a bottleneck feed-forward network) trained to adapt a frozen base model to a single, well-defined downstream task, such as named entity recognition or natural language inference.

• Purpose: Captures the parametric knowledge required for that specific task.
• Property: It is modular and composable; it can be plugged in/out or combined with other adapters.
• Prerequisite for Composition: Techniques like AdapterFusion and Adapter Merging require a library of pre-trained, high-quality task-specific adapters as their source of knowledge.

A Domain Adapter is trained to adapt a base model to the linguistic and stylistic patterns of a specific data domain, such as biomedical literature, legal contracts, or technical support dialogues. Unlike a task adapter, it focuses on domain shift rather than a new prediction head.

• Composition Role: Can be combined with a task adapter (e.g., for sentiment analysis) to perform that task within the specialized domain.
• Stacking: Often used in sequential composition, where a domain adapter is applied first to adjust the representation space, followed by a task adapter.
• Benefit: Enables efficient specialization for enterprise verticals without training a full model from scratch for each domain-task pair.