Target Modules

The most common and effective target modules for LoRA in transformer architectures. These linear layers project the input into query (Q), key (K), value (V), and output (O) representations within the multi-head self-attention mechanism.

• Why they are targeted: These projections are dense, parameter-rich layers directly responsible for modeling contextual relationships. Adapting them efficiently captures task-specific attention patterns.
• Typical Practice: Applying LoRA to just the query (Q) and value (V) projections is a standard, highly effective configuration, as popularized by the original LoRA paper for GPT models.

The two linear layers (often called the "up" and "down" projections) within the transformer's feed-forward network block. This module applies a position-wise non-linear transformation to each token's representation.

• Why they are targeted: The FFN layers contain a significant portion of a transformer's parameters and are hypothesized to function as key-value memory networks. Adapting them allows the model to adjust its internal "fact" storage and retrieval for a new domain.
• Consideration: Applying LoRA to FFN layers often yields substantial gains for knowledge-intensive tasks but increases the number of trainable parameters compared to targeting only attention projections.

The input token embedding matrix and the final language modeling head (output projection), which are often tied (share weights) in decoder-only LLMs.

• Why they are targeted: Adapting these layers allows the model to adjust the semantic meaning of tokens for the target domain, which is crucial for specialized vocabulary or terminology.
• Caveat: Fine-tuning these very large matrices (vocab_size × hidden_dim) with LoRA can be highly parameter-efficient but may have a smaller impact per parameter compared to adapting intermediate layers. It is often used in combination with other modules.

The query (Q) projection within the cross-attention layers of encoder-decoder models (e.g., T5, BART). This layer allows the decoder to attend to the encoder's output representations.

• Why they are targeted: For sequence-to-sequence tasks like translation or summarization, adapting the cross-attention mechanism is critical for learning task-specific alignment between source and target sequences.
• Application: In models like T5, applying LoRA to the cross-attention Q projection and the decoder's self-attention projections is a standard and effective strategy.

The gain and bias parameters of Layer Normalization (LayerNorm) layers. While not a traditional "projection," these are small, per-layer sets of parameters that control feature scaling and shifting.

• Why they are targeted: Fine-tuning LayerNorm parameters (sometimes called "LN tuning") is an extremely parameter-efficient PEFT method on its own. When combined with LoRA on linear layers, it provides a complementary adaptation mechanism for feature distribution.
• Efficiency: This adds only 2 × hidden_dim trainable parameters per adapted layer, making it a common supplement to LoRA for marginal cost.

The process of determining which layers to apply LoRA to, balancing performance, efficiency, and task needs.

• Rule-based Selection: Common heuristics include targeting all attention projections (Q, K, V, O) in specified transformer blocks (e.g., the last N layers).
• Automated Search: Methods like LoRAHub or Laplacian Singular Value analysis can be used to identify the most impactful layers for adaptation, moving beyond heuristic selection.
• Performance Trade-off: More target modules generally lead to better task performance but increase the number of trainable parameters and memory overhead. The optimal set is task- and model-dependent.

The attention mechanism is a neural network component that allows a model to dynamically focus on different parts of the input sequence. In transformers, this is implemented via self-attention heads. LoRA commonly targets specific linear projections within these heads—namely the query (Q), key (K), and value (V) matrices—as they are primary sites for task-specific adaptation. Injecting low-rank updates here allows the model to efficiently learn new attention patterns for downstream tasks.

A feed-forward network (FFN) is a fully connected layer within a transformer block that applies a non-linear transformation to each token's representation independently. In many architectures, the FFN contains a significant portion of the model's parameters. Applying LoRA to the linear layers inside the FFN (e.g., the up-projection and down-projection in a SwiGLU block) is a common strategy to adapt the model's internal feature representations efficiently for new domains.

A linear projection is a matrix multiplication operation that transforms an input vector from one dimension to another. In transformer architectures, these are ubiquitous:

• Query/Key/Value Projections in attention blocks.
• Output Projection in attention blocks.
• Up/Down Projections in feed-forward networks. These dense layers are the primary target modules for LoRA because their weight matrices are large and directly responsible for feature transformation. The low-rank update ΔW = BA is injected in parallel to these frozen projections.

A transformer block is the fundamental repeating unit in transformer-based models, typically consisting of a multi-head self-attention mechanism and a feed-forward network, with residual connections and layer normalization. When configuring LoRA, engineers specify which components within these blocks are target modules. Common patterns include targeting all linear layers in the attention mechanism, the FFN, or both. The choice directly impacts the number of trainable parameters and the adaptation's effectiveness.

Parameter-Efficient Fine-Tuning (PEFT) is a family of techniques designed to adapt large pre-trained models by updating only a small subset of parameters. LoRA is a prominent PEFT method. The concept of target modules is central to all PEFT strategies, as each method (e.g., prefix tuning, adapters) must define which parts of the model architecture will be modified. The efficiency gains of PEFT are realized by selectively applying updates to these critical modules rather than the entire network.

Model configuration in the context of LoRA refers to the hyperparameters and architectural choices that define how the adapters are integrated. Key configuration parameters include:

• target_modules: A list specifying which module types to apply LoRA to (e.g., ['q_proj', 'v_proj']).
• r (rank): The intrinsic dimension of the low-rank matrices.
• lora_alpha: The scaling factor for the low-rank update.
• lora_dropout: The dropout probability for regularization. Precise configuration of target_modules is essential for balancing performance, parameter efficiency, and training stability.