Parameter-Efficient Fine-Tuning (PEFT)

Low-Rank Adaptation (LoRA) freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture. It hypothesizes that weight updates during adaptation have a low 'intrinsic rank'. Instead of updating the full weight matrix (W \in \mathbb{R}^{d \times k}), LoRA represents the update with a low-rank decomposition (W + \Delta W = W + BA), where (B \in \mathbb{R}^{d \times r}), (A \in \mathbb{R}^{r \times k}), and the rank (r \ll \min(d, k)). Only A and B are trained, reducing trainable parameters by orders of magnitude (e.g., 0.1% of the original). This technique is highly effective for fine-tuning large language models and is a cornerstone of edge AI adaptation due to its minimal storage overhead for adapter weights.

Adapter layers are small, trainable neural network modules inserted between the layers of a frozen pre-trained model. A standard adapter consists of a down-projection to a lower dimension, a non-linearity, and an up-projection back to the original dimension, forming a bottleneck. Only these adapter parameters are updated during fine-tuning. Key variants include:

• Parallel Adapters: Process the input in parallel to the frozen layer.
• Serial Adapters: Placed sequentially after a frozen layer.
• Multi-Head Adapter: Shares adapter parameters across attention heads. This method is highly modular, allowing task-specific adapters to be swapped in and out, which is ideal for multi-task edge systems. The primary trade-off is a slight increase in inference latency due to the extra forward passes through the adapter modules.

These techniques prepend a sequence of trainable 'soft' prompt vectors to the model's input or hidden states, steering the model's behavior without modifying its core weights.

• Prompt Tuning: Adds trainable embeddings to the input layer. It is simple but can require many tokens for effectiveness.
• Prefix Tuning: Introduces trainable vectors (the prefix) to the hidden states at every layer of the Transformer, offering more expressive control. The trainable parameters are often re-parameterized via a small MLP to improve training stability. Both methods keep the base model completely frozen, making them extremely parameter-efficient. They are particularly suited for generative language tasks on the edge, as they add minimal overhead to the inference context length.

IA³ is a PEFT method that scales inner activations of a pre-trained model using learned vectors. Instead of adding new layers or matrices, it introduces three sets of small, task-specific vectors that multiply (scale) the keys, values, and feed-forward inner activations within Transformer blocks. This approach:

• Adds even fewer parameters than LoRA (often ~0.01% of the original model).
• Introduces no inference latency during deployment, as the scaling is fused into the base model weights via a simple element-wise multiplication.
• Is highly effective for multi-task learning, as different scaling vectors can be tuned for different tasks and combined. Its fusion-friendly nature makes it exceptionally well-suited for compiled edge deployments where every operation counts.

BitFit is a strikingly simple PEFT technique where only the bias terms within the model are tuned, while all other weights remain frozen. In Transformer models, these include biases in attention layers, feed-forward networks, and layer norms. Despite updating a tiny fraction of parameters (often <0.1%), BitFit can achieve competitive performance on many NLP tasks. Its advantages for edge AI are profound:

• Minimal Memory Footprint: Only the bias vectors need to be stored and updated.
• Training Efficiency: The optimizer state is extremely small.
• Deployment Simplicity: The tuned biases can be directly added to the frozen base model, requiring no architectural changes or conditional logic during inference. It represents a lower bound on parameter efficiency for fine-tuning.

Weight-Decomposed Low-Rank Adaptation (DoRA) is an advanced technique that builds upon LoRA. It decomposes a pre-trained weight matrix into a magnitude vector and a directional matrix. DoRA fine-tunes the direction component using LoRA while also learning a separate magnitude vector. This decomposition more effectively aligns the fine-tuned model's behavior with the full fine-tuning baseline. The process involves:

1. Decomposing the frozen weight (W_0) into magnitude (m) and direction (V).
2. Training a LoRA module (B A) to update the direction.
3. Learning a new magnitude vector (m'). The final weight is reconstructed as (W' = m' \cdot \frac{V + BA}{|V + BA|_c}). DoRA often outperforms standard LoRA with a comparable number of trainable parameters, providing a more powerful adaptation tool for critical edge applications where accuracy is paramount.

PEFT slashes the computational cost of adaptation. By training only 0.1%-10% of parameters, it requires far less GPU memory and training time compared to full fine-tuning.

• Memory: Enables fine-tuning of large models (e.g., 70B parameters) on consumer-grade hardware.
• Speed: Faster training cycles and lower cloud compute bills.
• Carbon Footprint: Significantly reduces the energy consumption associated with model adaptation, supporting sustainable AI development.

Quantization-Aware Training (QAT) is a model compression technique that prepares a network for efficient integer inference. Unlike post-training quantization, QAT simulates the effects of lower precision (e.g., INT8) during the fine-tuning or training process. This allows the model to learn parameters that are robust to the precision loss, typically recovering nearly all the accuracy drop seen with standard quantization.

• Process: 'Fake' quantization nodes are inserted into the model graph during training, rounding weights and activations to mimic integer arithmetic.
• Edge Synergy with PEFT: QAT can be combined with PEFT methods like LoRA, creating ultra-efficient models that are both task-adapted and hardware-optimized for edge deployment.

Structured Pruning is a model compression technique that removes entire structural components from a neural network to create a smaller, more efficient architecture. Unlike unstructured pruning, which creates irregular sparsity, structured pruning removes coherent units like entire channels, filters, or attention heads, resulting in a model that runs efficiently on standard hardware without specialized sparse kernels.

• Common Targets: Pruning less important channels in convolutional layers or heads in Transformer layers.
• Application with PEFT: Can be applied before or in conjunction with PEFT. For example, a model can first be pruned to reduce its base size, then efficiently fine-tuned for a new task using LoRA, maximizing edge efficiency.

Knowledge Distillation is a compression paradigm where a large, accurate teacher model transfers its knowledge to a smaller, more efficient student model. The student is trained not just on hard labels, but to mimic the teacher's softened output distributions (logits) or intermediate feature representations. This often allows the student to surpass the performance of a model trained only on the original data.

• Relation to PEFT: While PEFT adapts a large model in-place, distillation creates a separate, smaller model. The techniques are complementary: a large model fine-tuned via PEFT can serve as an excellent teacher for distilling a tiny model suitable for the most constrained edge devices.

Memory Footprint is a critical constraint for edge AI, referring to the total amount of memory (RAM and storage) required to load and run a model. It encompasses the model parameters, activations (intermediate layer outputs), and system buffers. PEFT techniques directly optimize this metric by drastically reducing the number of trainable parameters that must be stored during adaptation and by enabling the use of smaller, quantized base models.

• Key Components:
  • Parameter Memory: Size of the model weights. Reduced by PEFT adapters and quantization.
  • Activation Memory: Often the bottleneck for inference with large batch sizes or sequence lengths. Addressed through activation compression techniques.
• Edge Impact: Lower memory footprint enables deployment on devices with stringent RAM limits, such as microcontrollers and mobile phones.

Inference Latency is the time delay between submitting an input to a model and receiving its output, measured in milliseconds. It is a primary performance metric for real-time edge applications. While PEFT itself reduces training costs, its outputs—smaller adapter weights—are often fused back into the base model for inference, resulting in no direct latency penalty. The real latency benefits come from combining PEFT with other compression techniques it enables.

• Drivers of Latency: Model size (FLOPs), memory bandwidth, hardware parallelism, and kernel efficiency.
• Compression Synergy: PEFT facilitates the use of heavily quantized or pruned base models, which directly reduce computational FLOPs and memory bandwidth requirements, leading to lower latency. A model fine-tuned with LoRA on a pruned, INT8-quantized base is optimized for minimal edge inference time.