ViT Adapters

The defining characteristic of a ViT Adapter is its insertion into a frozen backbone Vision Transformer. The core pre-trained weights of the ViT remain completely locked, preserving its general visual knowledge. The adapter modules are injected at specific injection points (e.g., after the Multi-Head Self-Attention or MLP blocks) and only these small modules are trained. This creates a clear separation between the foundational model and the task-specific delta weights.

ViT Adapters employ a bottleneck design to minimize trainable parameters. A standard adapter consists of a down-projection layer (to a smaller bottleneck dimension), a non-linearity (e.g., GELU), and an up-projection layer back to the original feature dimension. This design ensures the adapter's parameter count is a small fraction of the backbone's, often controlled by a reduction factor (e.g., 16 or 32), making adaptation highly parameter-efficient.

Unlike language model adapters that process 1D sequences, ViT Adapters must handle 2D spatial feature maps. They are designed to process and modify the spatial structure of visual tokens. This is critical for dense prediction tasks like semantic segmentation or object detection, where the adapter must learn to refine spatial representations and capture task-specific contextual relationships between image patches.

ViT Adapters enable efficient multi-task learning and continual learning. Multiple task-specific adapters can be trained independently on a single frozen backbone. For a new input, the relevant adapter can be activated. Techniques like AdapterFusion can be employed to learn to combine knowledge from multiple pre-trained adapters. This modularity prevents catastrophic forgetting when learning new tasks sequentially.

A primary use case for ViT Adapters is adapting image classification models (e.g., ImageNet-pretrained ViTs) for dense prediction. The adapters transform the backbone's features to be suitable for task-specific decoder heads, such as U-Net-like architectures for segmentation or Feature Pyramid Networks (FPNs) for detection. The adapter effectively bridges the generic pre-trained features and the specialized output head.

Performance depends on where and how many adapters are inserted. Common strategies include:

• Insertion after every Transformer block for maximum adaptability.
• Selective insertion in higher, more semantic layers for task-specific tuning.
• Using AdapterDrop to skip adapters in lower layers during inference for speed. The scaling of the bottleneck dimension and the use of parallel vs. sequential adapter placement are key hyperparameters for balancing capacity and efficiency.

An adapter is a small, trainable neural network module inserted into the layers of a frozen pre-trained model. It learns task-specific transformations of the intermediate activations, enabling efficient adaptation. Key characteristics include:

• Bottleneck Architecture: Typically uses a down-projection, non-linearity, and up-projection to limit parameters.
• Modularity: Can be inserted after attention or feed-forward layers in transformers.
• Reusability: Trained adapters can be composed or swapped for multi-task learning.

A visual adapter is a parameter-efficient module designed for insertion into vision transformers (ViTs) or convolutional neural networks (CNNs). It adapts a pre-trained visual backbone for new image tasks like segmentation or detection. Implementation involves:

• Spatial Processing: Often uses convolutional layers to handle 2D feature maps from ViT patches.
• Task Heads: Connects to downstream heads (e.g., mask decoder for segmentation).
• Efficiency: Updates <5% of total parameters compared to full fine-tuning.

Low-Rank Adaptation (LoRA) is a foundational PEFT method that approximates a model's weight update via low-rank matrices. For a pre-trained weight matrix W, it adds W + BA, where B and A are low-rank trainable matrices. Key attributes:

• Rank (r): The intrinsic dimension controlling adapter capacity (e.g., r=8).
• Additive Operation: The low-rank update is added to the frozen weight, avoiding inference latency.
• Versatility: Originally for language models, now applied to vision model attention blocks.

AdapterFusion is a two-stage, knowledge-composition technique for adapter-based models. It first trains multiple task-specific adapters independently on different datasets. A second-stage fusion layer is then trained to dynamically combine these adapters' outputs for a new target task. This enables:

• Transfer Learning: Leverages knowledge from multiple source tasks without catastrophic forgetting.
• Dynamic Routing: The fusion layer learns attention over adapter outputs.
• Efficiency: Avoids training a new adapter from scratch for every new task.

Injection points are the specific architectural locations within a neural network where PEFT modules are inserted. For Vision Transformers, common injection points include:

• Post-Attention: After the multi-head self-attention module, before the residual connection.
• Post-Feed-Forward: After the MLP block.
• Parallel Configuration: Where the adapter runs in parallel to the original layer, summing outputs. The choice of injection point significantly affects adaptation performance, computational overhead, and gradient flow.

The frozen backbone refers to the large, pre-trained base model (e.g., a ViT-L/16) whose original parameters are kept fixed (non-trainable) during parameter-efficient fine-tuning. This is the core efficiency premise of PEFT:

• Preserves General Knowledge: The model's foundational representations from pre-training on massive datasets (e.g., ImageNet-21k) remain intact.
• Reduces Memory Footprint: Only the small adapter parameters require gradient computation, drastically lowering GPU memory for training.
• Enables Rapid Adaptation: Multiple lightweight adapters can be trained for different tasks using the same frozen backbone.