AdapterHub

The system that defines how adapters are trained and combined. It supports the full spectrum of adapter-based fine-tuning methodologies.

• Training Protocols: Supports Adapter Tuning (training a single adapter), Multi-Task Training (training adapters for several tasks concurrently), and AdapterFusion (learning to combine pre-trained task adapters).
• Composition Strategies: Manages the sequential stacking (AdapterStacking) or parallel activation (Mixture-of-Adapters) of multiple adapter modules.
• Efficient Configuration: Allows fine-grained control over which model components are frozen and which adapters are active, minimizing computational overhead.

A structured classification system that organizes adapters by the base model architecture, the target task, and the data domain. This taxonomy is crucial for discoverability and interoperability.

• Model Architecture: Categories adapters by the base model they plug into (e.g., bert-base-uncased, roberta-large).
• Task Type: Classifies adapters according to the NLP task they solve, such as text classification, question answering, or sequence labeling.
• Domain/Language: Tags adapters with specific domains (e.g., medical, legal) or languages (e.g., de, zh), enabling precise reuse for specialized applications.

An adapter is a small, trainable neural network module inserted into a frozen pre-trained model to efficiently adapt it to a new task or domain. It typically consists of a down-projection, a nonlinear activation, and an up-projection, forming a bottleneck structure that minimizes added parameters. By training only the adapter's weights, the original model's knowledge is preserved while enabling task-specific adaptation with a fraction of the compute cost of full fine-tuning.

AdapterFusion is a knowledge composition technique that combines multiple pre-trained, task-specific adapters to solve a new task. It operates in two stages:

• Knowledge Extraction: Individual adapters are trained on diverse source tasks.
• Knowledge Composition: A new fusion layer is trained to learn a weighted combination of the frozen adapters' outputs. This enables transfer learning across tasks without catastrophic forgetting and allows a single model to leverage expertise from multiple domains, such as combining sentiment analysis and natural language inference adapters for a complex reasoning task.

Mixture-of-Adapters (MoA) is a sparse, conditional computation architecture inspired by mixture-of-experts. Multiple adapter modules are made available within a model, and a learned routing network dynamically selects a subset of adapters to activate for each input. This enables:

• Specialization: Different adapters can become experts for different input types or sub-tasks.
• Scalability: The total parameter count grows, but only a fraction is used per inference, keeping computational cost manageable.
• Multi-Task Efficiency: A single MoA-augmented model can effectively handle a broad distribution of tasks by routing to the appropriate expert adapters.

Adapter merging is a post-training technique to combine multiple task-specific adapters into a single, multi-capable adapter without additional training. Common methods include:

• Weight Averaging: Simple arithmetic mean of adapter parameters.
• Task Arithmetic: Linear combinations with scaling factors.
• RegMean: A more advanced method that considers the covariance of activations. This creates a merged adapter capable of performing well on all source tasks, enabling efficient multi-task inference without the need to switch or compose adapters dynamically, thus reducing system complexity.

Adapter quantization is a model compression technique that reduces the numerical precision of an adapter's weights and activations to decrease memory footprint and accelerate inference. Common practices include:

• Post-Training Quantization (PTQ): Converting a trained full-precision (FP32) adapter to a lower format like INT8 after training.
• Quantization-Aware Training (QAT): Simulating quantization during adapter training to improve final low-precision performance. Since adapters represent a small fraction of total model parameters, quantizing them can yield significant relative memory savings for the trainable components, making deployment on edge devices more feasible.

Adapter inference overhead refers to the additional latency and compute cost incurred during a forward pass due to the inserted adapter modules. Key factors include:

• Adapter Size: The bottleneck dimension directly impacts FLOPs.
• Placement Strategy: Parallel adapters (Houlsby) add more overhead than serial adapters (Pfeiffer).
• Hardware: The overhead is more pronounced on hardware without optimized kernels for the small, dense projections in adapters. Techniques like AdapterDrop (skipping adapters in lower layers) and dynamic routing (MoA) are used to mitigate this overhead, trading a small amount of performance for significantly faster inference.