Serial Adapter

A serial adapter processes the model's internal activations in a strict sequential order. It is inserted directly into the forward pass of a transformer block, typically after the feed-forward network (FFN). The input to the adapter is the output of the preceding layer, and its output is passed to the subsequent layer (e.g., a layer normalization or the next block). This creates a clear, linear data flow: FFN → Adapter → Next Layer. This contrasts with parallel adapters, which process activations concurrently with the main FFN path.

The core of a serial adapter is a projection bottleneck designed to minimize parameters. Its standard structure is:

• Down-Projection: A linear layer that projects the high-dimensional input (e.g., d_model=768) down to a much smaller bottleneck dimension (e.g., d_bottleneck=64).
• Non-Linear Activation: A function like GELU or ReLU applied to the compressed representation.
• Up-Projection: A linear layer that projects back up to the original input dimension. This d_model → d_bottleneck → d_model design ensures the number of added trainable parameters is proportional to 2 * d_model * d_bottleneck + d_bottleneck, which is typically <1% of the base model's parameters.

To ensure stable training and prevent degradation of the pre-trained model's knowledge, serial adapters integrate via a residual connection. The adapter's output is added to the original input activation (the skip connection). Formally, if x is the input to the adapter module A, the output is x + A(x). This design guarantees that at initialization (when A(x) ≈ 0), the network behaves identically to the frozen base model. The gradient can flow through both the adapter and the skip connection, which mitigates vanishing gradients and allows the adapter to learn a parameter-efficient delta to the original function.

While serial adapters introduce a fixed computational overhead, it is highly predictable and relatively small. The overhead is primarily the cost of the two matrix multiplications (down/up-projection) and the activation function. For a transformer block, adding a serial adapter typically increases inference latency by 5-20% per adapted block, depending on the bottleneck dimension. This is a key trade-off: serial adapters are less parameter-efficient than methods like LoRA (which adds zero inference latency) but often achieve higher task performance due to their more expressive, non-linear transformation.

Two canonical configurations define where serial adapters are placed within a transformer block:

• Pfeiffer Adapter: A simplified, efficient architecture that inserts a single adapter layer only after the FFN. This is the most common serial adapter configuration, balancing performance and efficiency.
• Houlsby Adapter: The original, more expressive architecture that inserts two adapter layers per block: one after the multi-head attention (MHA) and one after the FFN. While more powerful, it doubles the parameter count and inference overhead compared to the Pfeiffer configuration. The choice between them is an efficiency-performance trade-off studied in adapter design.

The standardized interface of serial adapters makes them ideal modules for composition. Techniques like AdapterFusion learn to combine outputs from multiple pre-trained, task-specific serial adapters. Mixture-of-Adapters (MoA) systems use a router to dynamically select which serial adapter to activate. Adapter stacking involves sequentially connecting multiple adapters (e.g., a language adapter followed by a task adapter) within the same block. This modularity enables complex behaviors like multi-task learning and cross-lingual transfer without catastrophic interference, as each adapter's function is isolated and composable.

A serial adapter is a small, trainable neural network module placed in sequence with a pre-existing layer, typically the feed-forward network (FFN) within a transformer block. Its standard design is a bottleneck architecture:

• Down-Projection: A linear layer reduces the input dimension (e.g., 768) to a smaller bottleneck size (e.g., 48).
• Non-Linear Activation: A function like ReLU or GELU introduces non-linearity.
• Up-Projection: A linear layer projects back to the original dimension. The output of this adapter is added to the original FFN's output via a residual connection, allowing the model to learn task-specific adjustments without modifying the frozen base weights.

The Pfeiffer adapter is the canonical serial adapter design. It simplifies the earlier Houlsby architecture by inserting a single adapter layer only after the feed-forward network in each transformer block. This placement is highly effective for task adaptation and has become a standard due to its efficiency.

Key Characteristics:

• Insertion Point: After the FFN, before the final residual connection.
• Parameters: Typically adds 0.5% - 8% new parameters relative to the base model.
• Advantage: Simpler, faster to train, and often matches the performance of more complex two-adapter-per-block designs.

The original Houlsby adapter architecture inserts two serial adapters per transformer block: one after the multi-head attention (MHA) module and one after the feed-forward network (FFN). Both adapters use the bottleneck design and residual connections.

Design Rationale:

• The first adapter modifies attention outputs.
• The second adapter modifies feed-forward outputs.
• This provides finer-grained control over the transformation of activations at two key stages within the block. While more expressive, it introduces roughly double the parameters and latency overhead compared to the Pfeiffer variant.

The primary architectural distinction in adapter design is serial versus parallel placement relative to the base layer.

Serial Adapter:

• Operation: output = BaseLayer(x) + Adapter(BaseLayer(x))
• Processes the output of the frozen base layer.
• Alters the post-activation signal.

Parallel Adapter:

• Operation: output = BaseLayer(x) + Adapter(x)
• Processes the input to the frozen base layer in parallel.
• Provides a complementary signal to the base layer's output.

Serial adapters are more common in NLP, while parallel adapters are often found in vision transformers. Serial placement generally provides more direct task-specific modulation of layer outputs.

Serial adapters introduce a predictable computational trade-off.

Parameter Efficiency:

• A bottleneck dimension of 64 on a 768D model adds ~0.5M trainable parameters per adapter layer.
• For a 7B parameter model with adapters in every block, this represents < 1% of the total parameter count.

Inference Overhead:

• Latency: Adds 10-20% overhead compared to the base model, as it introduces two extra matrix multiplications (down/up projection) per adapted block.
• Memory: Requires storing the small adapter weights and the intermediate bottleneck activation.

Optimization: Techniques like AdapterDrop (skipping adapters in early layers during inference) and adapter quantization can reduce this overhead significantly.

An adapter is a small, trainable neural network module inserted into a frozen pre-trained model to efficiently adapt it to a new task or domain. It acts as a plug-in, allowing the model to learn new capabilities while preserving its original, general-purpose knowledge. Adapters are the fundamental building block for parameter-efficient fine-tuning (PEFT).

• Core Function: Learn a task-specific transformation of the model's internal activations.
• Key Benefit: Enables adaptation with only 1-5% of the original model's parameters.
• Typical Use: Adapting a large language model like Llama 3 for a specialized task like legal document analysis.

A parallel adapter is an adapter module that operates in parallel with a pre-trained layer's feed-forward network, adding its output to the main activation path via a residual connection. This contrasts with a serial adapter, which is placed sequentially in the data flow.

• Architecture: The adapter's computation path runs alongside the frozen feed-forward layer.
• Residual Connection: The adapter's output is added to the original layer's output: output = FFN(x) + Adapter(x).
• Advantage: Can be more computationally efficient than serial placement as it doesn't lengthen the critical path.

Adapter tuning is the process of fine-tuning a pre-trained model by training only the parameters of inserted adapter modules while keeping the original model weights frozen. This is the core training procedure for all adapter-based methods.

• Training Objective: Minimize loss on a downstream task by updating only adapter weights.
• Frozen Base Model: The original, massive parameter set remains unchanged, preventing catastrophic forgetting.
• Workflow: Insert adapters, freeze the base model, then train on new task data.

A bottleneck adapter is a specific adapter architecture that uses a low-dimensional bottleneck layer to project activations into a smaller space and back, minimizing the number of trainable parameters. It is the most common adapter design.

• Standard Structure: Down-Projection → Non-linearity (e.g., ReLU) → Up-Projection.
• Parameter Efficiency: The bottleneck dimension (e.g., 64) is much smaller than the model's hidden dimension (e.g., 4096), creating the efficiency gain.
• Example: In a transformer with a 1024-dimensional hidden state, a bottleneck adapter might project down to 64 dimensions, then back up to 1024.

AdapterFusion is a technique that combines knowledge from multiple pre-trained, task-specific adapters by learning a weighted composition of their outputs to perform a new, composite task. It enables knowledge transfer without forgetting previous adaptations.

• Two-Stage Process: First, train individual adapters on separate source tasks (e.g., sentiment, NER). Second, freeze those adapters and train a small fusion layer to combine them for a target task.
• Key Benefit: Avoids negative interference between tasks and leverages diverse expertise.
• Use Case: Creating a model that can answer complex questions requiring both sentiment understanding and entity recognition.