Injection Points

Injection points are not arbitrary; they are strategically chosen based on the model's computational graph and information flow. Common locations in transformer architectures include:

• After the multi-head attention module, to modify contextual representations.
• After the feed-forward network (FFN), to adapt processed features.
• Within cross-attention layers in encoder-decoder or multimodal models, to align information between modalities. Placing modules at these points allows the injected parameters to intercept and transform the activations that carry the most task-relevant information.

A core principle of injection points is non-destructive interference with the frozen backbone. The inserted module (e.g., an adapter) processes the incoming activation and adds its output, creating a residual connection. The operation is typically: y = x + f(x), where x is the original activation and f is the injected module. This ensures the original, general-purpose knowledge encoded in the pre-trained weights remains intact and accessible, while the module learns a task-specific delta.

To maintain parameter efficiency, modules at injection points almost always employ a bottleneck architecture. For an adapter, this involves:

• A down-projection to a lower-dimensional space (e.g., 64 or 128 dimensions).
• A non-linearity (e.g., GELU).
• An up-projection back to the original feature dimension. The bottleneck dimension (or reduction factor r) is the critical hyperparameter, trading off adapter capacity against the number of new parameters. A common setting is r=16, reducing parameters by ~0.5% of the original model.

The choice of injection point granularity significantly impacts performance and efficiency.

• Layer-wise Injection: A module is inserted at a chosen point in every transformer block (e.g., after every FFN). This provides comprehensive adaptation but higher parameter cost.
• Selective Injection: Modules are inserted only into a subset of layers, often based on heuristics (e.g., only the top 10 layers) or learned importance (as in AdapterDrop). This reduces compute during inference.
• Component-wise Injection: In methods like IA³, lightweight scaling vectors are injected at multiple points within a single layer (e.g., for keys, values, and FFN outputs), offering very fine-grained control.

The optimal injection point varies by model architecture and data modality.

• Encoder Models (e.g., BERT): Injections are typically placed within the encoder stack. BERT Adapters commonly insert modules after the FFN layer of each transformer block for NLP understanding tasks.
• Vision Models (e.g., ViT): ViT Adapters may inject at different stages of the visual backbone, sometimes incorporating spatial feature maps for dense prediction tasks like segmentation.
• Multimodal Models (e.g., CLIP, BLIP): VL-Adapters or Cross-Modal Adapters require injection points that affect the fusion mechanisms between modalities, such as within cross-attention layers or at the output of unimodal encoders before fusion.

While parameter-efficient, injection points add computational overhead to the model's forward pass. The primary costs are:

• The extra matrix multiplications of the injected module.
• The memory bandwidth required to load the additional parameters and intermediate activations. Techniques like AdapterDrop explicitly address this by dynamically pruning adapters from lower, less critical layers during inference. The design of the injection point and module (e.g., using a smaller bottleneck dimension) is a direct trade-off between adaptation quality and inference speed.

A small, trainable neural network module inserted at specific injection points within a frozen pre-trained model. It learns task-specific transformations of the intermediate activations flowing through the network.

• Typically consists of a down-projection, non-linearity, and up-projection.
• The bottleneck dimension controls its capacity and parameter efficiency.
• Enables efficient adaptation by updating only the adapter parameters, leaving the original model weights frozen.

The large, pre-trained base model (e.g., BERT, ViT, CLIP) whose parameters are kept fixed and non-trainable during parameter-efficient fine-tuning.

• Serves as a stable feature extractor with rich prior knowledge.
• All adaptation occurs via small, added modules (like adapters or prefixes) at the injection points.
• This is the core premise of PEFT: preserving the backbone's general capabilities while learning task-specific behaviors efficiently.

The small set of learned parameter changes (Δ) that represent the task-specific adaptation applied to a frozen pre-trained model during PEFT.

• In methods like LoRA, these are the low-rank matrices added to the original weights.
• The collection of all delta weights from a fine-tuning run forms a Task Vector.
• This conceptualization enables powerful techniques like model merging, where deltas from multiple tasks are combined arithmetically.

The application of parameter-efficient fine-tuning techniques specifically to encoder-only transformer models like BERT, RoBERTa, or DeBERTa.

• Injection points are typically after the feed-forward network or attention output within each transformer layer.
• Common tasks include text classification, named entity recognition (NER), and question answering.
• BERT Adapters are a canonical example, demonstrating efficient adaptation for natural language understanding.

Using parameter-efficient methods to adapt the fusion mechanisms in pre-trained multimodal models (e.g., CLIP, BLIP).

• Injection points target layers responsible for aligning and combining information from different modalities (text, image, audio).
• VL-Adapters and Cross-Modal Adapters are modules designed for these fusion points.
• Enables efficient adaptation to downstream tasks like visual question answering (VQA) or image captioning without retraining the entire complex model.

A key hyperparameter in adapter-based PEFT that defines the size of the adapter's hidden layer, controlling its capacity and parameter count.

• It creates a computational bottleneck, first projecting activations down to this dimension and then back up.
• A smaller bottleneck dimension increases parameter efficiency but may reduce adapter capacity.
• It is often set via a reduction factor (e.g., reducing 768-dim activations to a bottleneck of 48, a factor of 16).