Pfeiffer Adapter

Unlike the original Houlsby adapter which inserts two adapter modules per transformer block, the Pfeiffer adapter inserts only one adapter layer, placed exclusively after the feed-forward network. This cuts the number of added parameters and sequential operations nearly in half, directly reducing adapter overhead during both training and inference.

• Location: Post feed-forward network within each transformer block.
• Impact: Reduces sequential depth and latency compared to dual-adapter designs.

The core of the Pfeiffer adapter is a bottleneck feed-forward network. It projects the input activation into a low-dimensional space, applies a non-linearity, and projects it back to the original dimension. This creates a powerful but parameter-efficient function.

• Structure: Down-projection → Non-linearity (e.g., GELU) → Up-projection.
• Bottleneck Dimension (d): A hyperparameter (e.g., 64, 128) that controls the adapter's capacity and parameter count. A smaller d increases efficiency but may reduce representational power.

The adapter is integrated into the transformer block via a residual connection. The output of the adapter is added to the output of the original feed-forward network. This ensures training stability and prevents degradation of the base model's pre-existing knowledge, as the adapter can learn to output near-zero values initially.

• Formula: output = FFN(x) + Adapter(FFN(x))
• Benefit: Enables stable training from a near-identity initialization and preserves the frozen base model's capabilities.

The Pfeiffer adapter exemplifies parameter-efficient fine-tuning (PEFT). Only the adapter's parameters (the down-projection, up-projection, and sometimes a layer norm) are trained, while the entire pre-trained transformer model remains frozen. This results in a tiny fraction of trainable parameters compared to full fine-tuning.

• Typical Budget: Often <1% of the base model's total parameters.
• Benefit: Dramatically reduces storage requirements (only the small adapter is saved per task) and mitigates catastrophic forgetting.

The Pfeiffer adapter operates in serial with the existing feed-forward network. The activation flows through the frozen FFN first, then through the trainable adapter. This is a key distinction from parallel adapters, which process the input to the FFN in parallel with the FFN itself.

• Flow: Input → Frozen FFN → Adapter → Output.
• Implication: Adds a sequential computational step, but its small bottleneck size minimizes the latency impact.

A primary use case is training task-specific adapters. A unique Pfeiffer adapter can be trained for each new downstream task (e.g., sentiment analysis, named entity recognition). These lightweight adapters can be swapped in and out atop the single, shared frozen base model, enabling efficient multi-task serving.

• Workflow: Freeze base model (e.g., BERT, GPT). Train only the adapter on Task A data. Save adapter A. Repeat for Task B with a new adapter.
• Tooling: Frameworks like AdapterHub are built to manage, share, and load these small task-specific modules.

An adapter is a small, trainable neural network module inserted into a frozen pre-trained model to efficiently adapt it to a new task or domain. It acts as a lightweight, plug-in extension that modifies the model's internal representations without altering the original, massive parameter set. This enables parameter-efficient fine-tuning (PEFT) by updating only the adapter's weights, drastically reducing computational and storage costs compared to full model retraining.

Adapter tuning is the training process specific to adapter-based PEFT. During this process, the parameters of the inserted adapter modules are trained on a downstream task's dataset, while the weights of the original, much larger pre-trained model remain completely frozen. This creates a clear separation between the foundational knowledge in the base model and the newly acquired task-specific knowledge in the adapters, facilitating efficient multi-task learning and easy model sharing.

A bottleneck adapter is the most common architectural pattern for adapter modules. Its design enforces parameter efficiency through a constrained internal dimension. Its structure is:

• A down-projection layer that projects the input activation to a low-dimensional bottleneck.
• A non-linear activation function (e.g., ReLU).
• An up-projection layer that projects back to the original dimension.
• A residual connection adding the adapter's output to the original input. This bottleneck is the key to efficiency; for example, reducing a 768-dimensional activation to a bottleneck of 64 creates an adapter with only ~0.1M trainable parameters.

The Houlsby adapter is the seminal architecture introduced in the 2019 paper "Parameter-Efficient Transfer Learning for NLP." It inserts two bottleneck adapter layers into each transformer block:

• One adapter placed after the multi-head attention sub-layer.
• A second adapter placed after the feed-forward network sub-layer. This design was the baseline against which later, more efficient variants like the Pfeiffer adapter were compared. The Pfeiffer adapter simplifies this by using only a single adapter per block, placed after the feed-forward network.

AdapterFusion is a two-stage knowledge composition technique that builds upon task-specific adapters.

• Stage 1 (Knowledge Extraction): Multiple independent adapters are trained on different source tasks.
• Stage 2 (Knowledge Composition): These pre-trained adapters are frozen, and a new, lightweight fusion layer is trained to learn a weighted combination of their outputs for a new target task. This allows a model to leverage diverse expertise (e.g., from sentiment, NER, and QA adapters) without catastrophic forgetting, enabling efficient transfer to complex, composite tasks.