Adapter

The seminal architecture proposed by Houlsby et al. in 2019. It inserts two adapter layers per transformer block: one after the multi-head attention sub-layer and another after the feed-forward network. Each adapter follows a bottleneck design (project-down, nonlinearity, project-up) and is integrated via a residual connection. This design provides comprehensive adaptation but introduces the highest parameter and compute overhead among classic adapter types.

A simplified, more efficient architecture proposed by Pfeiffer et al. It inserts a single adapter layer only after the feed-forward network within each transformer block. This placement is found to be nearly as effective as the dual-insertion Houlsby design while halving the number of trainable parameters and reducing inference latency. It has become a widely adopted default due to its favorable efficiency profile.

• Key Insight: The feed-forward network is the primary location for task-specific feature transformation.

An architectural variant where the adapter module operates in parallel with the existing feed-forward network (FFN), rather than in series. The adapter takes the same input as the FFN, processes it through its bottleneck, and its output is added to the FFN's output via a residual connection. This design can be more computationally efficient than serial adapters as it avoids increasing the critical path depth of the network.

• Example: The Adapter variant used in the adapter-transformers library often implements this parallel structure.

This refers not to placement, but to the internal structure common to most adapters. A bottleneck adapter reduces the activation dimension (e.g., from 768 to 64) via a down-projection matrix, applies a nonlinear activation (e.g., GELU), and then projects back to the original dimension with an up-projection matrix. The bottleneck dimension is the key hyperparameter, controlling the trade-off between adaptability (more parameters) and efficiency (fewer parameters).

• Core Formula: Adapter(x) = x + (Up(Activation(Down(x))))

A conditional computation architecture inspired by mixture-of-experts. Instead of one adapter per block, a set of adapter modules is made available. A lightweight, learned routing network (or gating function) dynamically selects which adapter(s) to activate for a given input. This allows the model to develop specialized adapters for different input types or subtasks, potentially increasing capacity and performance without a linear increase in active compute for every forward pass.

Specialized architectures designed to bridge different data modalities (e.g., vision, language, audio) within a frozen pre-trained model. These adapters often have modality-specific input projections to align heterogeneous data into a common latent space that the base model can process. Examples include:

• VL-Adapter: Injects trainable modules into a vision-language model (like CLIP) for downstream vision-language tasks.
• Audio Adapter: Adapts a language model to process audio spectrograms or features for speech tasks.

The most direct application is adapting a general-purpose model to a specific downstream task. A task-specific adapter is trained on a labeled dataset (e.g., for sentiment analysis, named entity recognition, or text classification) while the base model remains frozen. This allows a single foundational model to serve hundreds of specialized functions by swapping lightweight adapter weights, avoiding the need for separate full models.

• Example: Fine-tuning a BERT model for legal contract clause classification using an adapter.
• Benefit: Drastically reduces storage costs and enables rapid prototyping of new task capabilities.

Adapters excel at shifting a model's knowledge to a new data domain with distinct vocabulary, style, or factual knowledge. A domain adapter is trained on text from a specialized field (e.g., biomedical literature, financial reports, or technical documentation). This teaches the model the semantics and jargon of the target domain without catastrophic forgetting of its general knowledge.

• Example: Adapting T5 or GPT to generate coherent patient summaries from clinical notes.
• Key Mechanism: The adapter learns to 'translate' general language representations into domain-specific ones.

For multilingual base models (e.g., mBERT, XLM-R), language adapters provide a parameter-efficient path to improve performance on lower-resource languages. A separate adapter is trained per target language, allowing the model to develop stronger syntactic and semantic representations for that language.

• Example: Using AdapterHub to load a pre-trained German adapter for a multilingual question-answering system.
• Advantage: Enables scalable support for many languages without language-specific model copies, maintaining a shared core of cross-lingual knowledge.

Adapters enable modular multi-task systems. Individual adapters for different tasks or domains can be composed dynamically. AdapterFusion learns to combine outputs from multiple pre-trained adapters (e.g., sentiment + domain) for a new composite task. Adapter composition can involve stacking or parallel connections.

• Workflow: Train separate adapters for sentiment, toxicity detection, and legal domain. Use AdapterFusion to create a system for analyzing sentiment in toxic legal comments.
• Benefit: Achieves positive knowledge transfer between tasks while avoiding negative interference.

Adapters are ideal for continual learning scenarios where a model must learn new tasks sequentially without forgetting previous ones. Each new task gets its own task-specific adapter, which is stored. The frozen base model serves as a stable knowledge backbone, preventing catastrophic forgetting.

• Process: Train Adapter A for Task 1, freeze it. Train Adapter B for Task 2, freeze it. At inference, the correct adapter is activated per task.
• Enterprise Value: Allows safe, incremental model expansion over time (e.g., adding new product categorization schemas).

Multimodal adapters are used to bridge pre-trained unimodal models (e.g., a vision encoder and a language model) or to adapt large vision-language models to new tasks. They project features from one modality into the alignment space of another with minimal trainable parameters.

• Architecture: Often involves cross-attention layers or linear projections inserted between frozen visual and linguistic towers.
• Example: Efficiently fine-tuning CLIP for specialized medical image captioning by training only adapter layers that connect its image and text encoders to the new domain.

The core architectural component of an adapter. It is a small, self-contained neural network module, typically structured as a bottleneck with a down-projection, a nonlinear activation (e.g., ReLU), and an up-projection. This design projects the input activation into a lower-dimensional space and back, introducing task-specific adaptation with minimal new parameters. It is inserted into a pre-trained model's transformer blocks.

The specific training process for adapter-based PEFT. During adapter tuning, the parameters of the original, massive pre-trained model are completely frozen. Only the weights of the newly inserted adapter modules are updated via gradient descent. This isolates the learning to the small adapter parameters, making the process highly parameter-efficient and preventing catastrophic forgetting of the base model's knowledge.

A knowledge composition technique that leverages multiple pre-trained, task-specific adapters. Instead of training a new adapter from scratch, AdapterFusion learns to combine the outputs of existing adapters via a learned attention-based weighting mechanism. This allows a model to perform a new task by dynamically fusing knowledge from adapters trained on related tasks (e.g., sentiment, emotion, toxicity).

An architecture inspired by mixture-of-experts, where multiple adapter modules are available within a model. A learned routing network (e.g., a gating function) dynamically selects which subset of adapters to activate for a given input. This enables conditional computation and can model complex task distributions more efficiently than a single monolithic adapter, as different "expert" adapters handle different input types.

A post-training technique to create a multi-task adapter without additional training. The weights of several single-task adapters (trained independently) are combined using simple arithmetic operations, such as task arithmetic (adding weighted task vectors) or averaging. The resulting merged adapter can perform all constituent tasks reasonably well, enabling efficient multi-task inference from a single adapted model.

A model compression technique applied specifically to adapter modules to reduce their deployment footprint. It involves reducing the numerical precision of the adapter's weights and activations—for example, from 32-bit floating-point (FP32) to 8-bit integers (INT8). This significantly decreases the memory usage and can accelerate inference latency on supported hardware, with a minimal impact on task performance.