AdapterDrop

AdapterDrop strategically removes adapters from the lower layers of a transformer model. Research indicates that adapters in the final layers contribute most to task performance, while those in early layers can be pruned. This is based on the observation that lower layers capture general features, and their adaptation is less critical for specific downstream tasks.

• Key Mechanism: During inference or training, the forward pass skips the adapter sub-layer in selected lower transformer blocks.
• Benefit: This directly reduces the number of matrix multiplications and non-linearities, decreasing FLOPs and latency.

The primary objective is to reduce the computational cost inherent to adapter-based models. Without AdapterDrop, each adapter adds two linear projections and an activation function per layer.

• Quantitative Impact: Pruning adapters from the bottom N layers can reduce total adapter compute by approximately (N / total_layers) * 100%. For a 12-layer model, dropping adapters from the first 6 layers can nearly halve the adapter-related computation.
• Result: This enables faster inference and more efficient multi-task serving, as the computational graph is simplified for a significant portion of the model's depth.

A core finding of AdapterDrop is that significant computational savings can be achieved with only a marginal drop in accuracy. The performance degradation is often non-linear and task-dependent.

• Empirical Observation: For many Natural Language Understanding (NLU) tasks, dropping adapters from up to two-thirds of the lower layers results in a performance loss of less than 1-2% relative to using all adapters.
• Implication: This creates a favorable efficiency-accuracy trade-off, making it a practical technique for production systems where latency and cost are critical constraints.

AdapterDrop can be applied in two primary configurations:

• Static AdapterDrop: A fixed set of lower-layer adapters is permanently removed after an analysis phase. This is optimal for stable deployment where the task and model are constant.
• Dynamic AdapterDrop: The system can selectively activate or deactivate adapters per input sample or based on a confidence threshold. This allows for adaptive computation, where simpler inputs bypass more adapters.
• Use Case: Dynamic pruning aligns with conditional computation paradigms, aiming to match computational cost to input complexity.

The technique is designed to work seamlessly with AdapterFusion and multi-adapter setups. In a stack of adapters (e.g., for multi-task learning), AdapterDrop can be applied uniformly across all parallel adapter paths in the pruned layers.

• Architecture Consideration: When using AdapterFusion, the fusion layer's input is modified, as it no longer receives outputs from the dropped lower-layer adapters. The fusion mechanism must be trained or adjusted to account for this pruned input space.
• Benefit: This maintains the parameter efficiency of the overall adapter-based system while adding a layer of computational efficiency.

AdapterDrop impacts both phases of the model lifecycle:

• Training: Can be used during fine-tuning to reduce GPU memory and training time. The gradients for the parameters in the dropped adapter modules are simply not computed.
• Inference: Offers direct latency reduction. The skipped adapter operations translate to faster forward passes, which is crucial for real-time applications and high-throughput serving environments.
• Deployment Advantage: The pruned model requires no special hardware or kernels; the efficiency gain comes from executing fewer operations in the standard computational graph.

An adapter is a small, trainable neural network module inserted into the layers of a frozen pre-trained model. It enables efficient adaptation to new tasks by learning task-specific transformations of the intermediate activations. Key characteristics include:

• Bottleneck Architecture: Typically uses a down-projection, non-linearity, and up-projection to reduce parameter count.
• Injection Points: Placed after the attention or feed-forward sub-layers within a transformer block.
• Frozen Backbone: The core model weights remain entirely fixed, preserving pre-trained knowledge and preventing catastrophic forgetting.

The bottleneck dimension is the size of the hidden layer within an adapter module, acting as the primary control for its capacity and parameter count. It is defined by a reduction factor (r) relative to the model's hidden dimension (d).

• Parameter Efficiency: The total adapter parameters scale as ~2 * d * r, where r << d.
• Trade-off: A smaller bottleneck increases efficiency but may limit task-specific learning capacity.
• Typical Values: Common reduction factors (r) range from 2 to 64, making adapters often <1% of the base model's parameters.

Injection points refer to the specific architectural locations within a neural network where parameter-efficient modules like adapters are inserted. For standard transformer models, common injection strategies are:

• Parallel Adapter: Injected in parallel to the feed-forward network, modifying activations via a residual connection.
• Sequential Adapter: Inserted sequentially after the feed-forward network or the multi-head attention module.
• Layer Selection: AdapterDrop specifically targets these points, deciding to remove adapters from lower transformer layers to skip their computation during inference.

A frozen backbone is the large, pre-trained base model (e.g., BERT, ViT, GPT) whose parameters are kept entirely fixed during parameter-efficient fine-tuning. This is a foundational principle of PEFT.

• Core Benefit: Preserves the model's general knowledge and representations learned during massive pre-training.
• Efficiency: Eliminates the memory and compute cost of backpropagating through the entire network.
• Stability: Prevents catastrophic forgetting of pre-trained skills. Only the small, added adapter parameters (or other PEFT modules) are updated.

In PEFT, trainable parameters refer to the tiny subset of a model's total weights that are updated during fine-tuning. For adapter-based methods, this includes:

• Adapter Weights: The matrices within the down-projection, non-linearity, and up-projection layers.
• LayerNorm Parameters: Sometimes the gain and bias parameters of layer normalization are also fine-tuned.
• Scale vs. Full Fine-Tuning: A model with 1B parameters might have only 2-10 million trainable parameters with adapters, compared to updating all 1B parameters in full fine-tuning.

Delta weights (ΔW) represent the small set of learned parameter changes applied to a frozen pre-trained model during PEFT. They encapsulate the task-specific adaptation.

• Mathematical Representation: The effective weight for a layer becomes W_effective = W_pretrained + ΔW.
• Adapter as Delta: An adapter module implicitly learns a delta transformation on the activations, not directly on the weights.
• Model Merging: Delta weights from multiple tasks can be arithmetically combined (e.g., added) to create a multi-task model, a key advantage of the PEFT paradigm.