Adapter Pruning

Adapter pruning induces sparsity within the adapter's weight matrices, setting a significant portion of parameters to zero. This is often achieved via magnitude-based pruning, where weights with the smallest absolute values are removed, or through structured pruning that removes entire rows, columns, or blocks of the weight matrix. The resulting sparse matrices can be stored and computed efficiently using specialized libraries, drastically reducing the adapter's inference overhead.

Pruning can be applied at different levels of granularity:

• Weight-level Pruning: Individual weights across the adapter's layers are pruned, offering the finest granularity and highest potential sparsity.
• Neuron/Filter-level Pruning: Entire neurons in the down-projection or up-projection layers are removed. This is a form of structured pruning that leads to direct reductions in matrix dimensions and is more amenable to hardware acceleration.
• Adapter-layer Pruning: Entire adapter modules are removed from specific transformer blocks, a strategy related to AdapterDrop. This offers the most significant latency reduction by bypassing computation in selected layers entirely.

The strategy for when and how to prune is critical. Common schedules include:

• One-shot Pruning: The adapter is trained to convergence, then pruned in a single step based on a saliency criterion (e.g., weight magnitude).
• Iterative Pruning: Also known as gradual pruning, this involves repeatedly training, pruning a small percentage of weights, and then fine-tuning the remaining sparse network. This often yields better performance than one-shot pruning at high sparsity levels.
• Pruning-Aware Training: The pruning process is integrated into the adapter's initial training loop, often using techniques like Lottery Ticket Hypothesis-inspired methods to identify and train a sparse subnetwork from the start.

The core challenge is balancing size reduction with task performance. The relationship is non-linear; initial pruning often has minimal impact, but beyond a sparsity threshold, accuracy degrades rapidly. Techniques to mitigate loss include:

• Iterative Pruning with Fine-tuning: Allows the network to recover after each pruning step.
• Regularization during Training: Applying L1 regularization encourages weights toward zero, making them better candidates for pruning.
• Importance Scoring: Using more sophisticated metrics than weight magnitude, such as the gradient or effect on the loss function, to identify which parameters are truly redundant.

Adapter pruning is highly complementary to adapter quantization. A common pipeline is Prune-Then-Quantize:

1. Train a dense adapter.
2. Prune it to a target sparsity and fine-tune.
3. Apply post-training quantization (e.g., to 8-bit integers) to the remaining non-zero weights. This combined approach can achieve multiplicative reductions in model size. Pruning creates sparse matrices, and quantization reduces the bit-width of each stored value, leading to extremely compact adapter representations suitable for edge deployment.

The pruning strategy can be tailored to the use case:

• Task-Agnostic Pruning: Prunes the adapter based on generic criteria (like weight magnitude) without explicit regard for the downstream task. This is faster but may remove task-critical features.
• Task-Specific Pruning: The pruning process is guided by the adapter's performance on the validation set of the target task. This can involve first-order (gradient-based) or second-order (Hessian-based) importance estimation to preserve parameters most relevant to the task's loss function, typically yielding better performance retention at high sparsity.

An adapter is a small, trainable neural network module inserted into a frozen pre-trained model to efficiently adapt it to a new task or domain. It acts as a lightweight, plug-and-play extension, allowing the base model's vast knowledge to be repurposed with minimal new parameters.

• Core component of adapter-based PEFT.
• Typically consists of a down-projection, nonlinear activation, and up-projection.
• Enables task specialization without catastrophic forgetting of the pre-trained base.

Adapter tuning is the process of fine-tuning a pre-trained model by training only the parameters of inserted adapter modules while keeping all original model weights frozen. This is the foundational training procedure that adapter pruning seeks to optimize post-hoc.

• Involves a small fraction (often <1%) of the model's total parameters.
• Dramatically reduces memory footprint and compute cost compared to full fine-tuning.
• The trained adapters are the target for subsequent pruning to reduce inference overhead.

AdapterDrop is a method for dynamically removing adapter layers from certain transformer blocks during inference to reduce computational latency with minimal performance loss. It is a form of adaptive inference and a close conceptual relative to pruning.

• Prunes adapters at the layer level rather than the parameter level.
• Can skip adapters in lower or middle layers based on task difficulty.
• Achieves significant speedups by reducing sequential operations in the model.

A bottleneck adapter is a specific, widely-used adapter architecture that uses a low-dimensional bottleneck layer to project activations into a smaller space and back. This design is the primary target for parameter-level pruning techniques.

• Architecture: d_model -> d_bottleneck -> d_model.
• The bottleneck dimension (d_bottleneck) is a key hyperparameter controlling size and a primary target for pruning.
• Its structured design makes it amenable to structured pruning methods that remove entire neurons or blocks.

Adapter quantization is the process of reducing the numerical precision of an adapter's weights and activations (e.g., from 32-bit floating-point to 8-bit integers) to decrease memory usage and accelerate inference. It is a complementary compression technique to pruning.

• Reduces the bit-width of parameters, whereas pruning reduces the number of parameters.
• Often applied after pruning for compounded efficiency gains.
• Enables efficient deployment on edge devices with limited memory bandwidth.

Adapter inference overhead refers to the additional computational cost and latency incurred during a forward pass due to the inserted adapter modules. This overhead is the core problem that adapter pruning aims to minimize.

• Composed of the extra matrix multiplications and nonlinearities in the adapter layers.
• Even small adapters can add 10-20% latency to transformer inference.
• Pruning directly targets this overhead by removing redundant computations within the adapter's own structure.