MCU-Compatible PEFT

MCUs typically have Static RAM (SRAM) measured in kilobytes, not gigabytes. This imposes a hard ceiling on the peak memory usage during both training and inference. MCU-Compatible PEFT must:

• Freeze the base model to keep its massive parameter count (often millions) in read-only flash memory.
• Design adapters with minimal parameter counts (e.g., low-rank matrices) whose gradients and optimizer states fit entirely in SRAM.
• Utilize static memory allocation at compile time to avoid the overhead and fragmentation of dynamic allocation.

MCU CPUs run at frequencies in the tens to hundreds of MHz, with no floating-point unit (FPU) or only a single-precision FPU. Compute-intensive operations are prohibitive. Techniques include:

• Leveraging quantized operations (INT8) which are faster and more energy-efficient than FP32.
• Simplifying adapter architectures (e.g., vanilla LoRA over more complex adapters) to reduce FLOPs per training step.
• Compiler-level optimizations like kernel fusion and efficient loop tiling to maximize hardware utilization.

Many MCUs are battery-powered or energy-harvesting, operating in milliwatt or microwatt regimes. The energy cost of training must be justified. This drives:

• Sparse training regimes where updates are triggered only by significant data drift, not continuously.
• Extremely short training loops with few epochs and small local batch sizes (often 1).
• Hardware-aware scheduling to leverage low-power sleep modes between training bursts.

Frameworks like PyTorch or TensorFlow are too large for MCUs. Deployment requires specialized toolchains:

• TinyML frameworks (e.g., TensorFlow Lite for Microcontrollers, Apache TVM) that support PEFT operator kernels.
• Ahead-of-Time (AOT) compilation that bakes the base model and adapter architecture into a single, optimized executable.
• Manual memory planning to map model tensors to specific SRAM addresses, avoiding runtime overhead.

MCU inference engines typically require a static graph defined at compile time. This conflicts with dynamic neural architectures. Solutions involve:

• Pre-defining adapter 'slots' in the base model graph where low-rank matrices can be injected.
• Designing for runtime adapter switching via pre-allocated memory buffers, not dynamic graph modification.
• Using hypernetworks that generate adapter weights from a small input, keeping the execution path fixed.

MCUs perform efficient integer arithmetic. Models must remain stable when quantized. MCU-Compatible PEFT is often quantization-aware from the start:

• Training the adapter in simulated INT8 (QAT) to learn to compensate for quantization noise.
• Using integer-friendly optimizers like straight-through estimators for gradient propagation.
• Ensuring adapter weights have low precision sensitivity, as they may be stored as INT8 or INT4.

BitFit is an extreme form of sparse fine-tuning where only the bias terms within a model are updated. For MCUs, this offers critical advantages:

• Minimal Parameter Overhead: Often less than 0.1% of total model parameters are trainable.
• Static Computation Graph: Since only biases are modified, the core matrix multiplication patterns remain unchanged, simplifying compiler optimizations.
• Efficient Storage: The delta (update) is a simple vector of biases, requiring minimal flash storage for deployment. It is highly effective for task adaptation when the primary need is to adjust feature activation thresholds.

This method optimizes a set of continuous virtual token embeddings prepended to the input sequence. MCU-compatible implementations focus on:

• Pre-allocation of Prompt Memory: A fixed buffer in SRAM is reserved to hold the trained prompt tensors.
• Compiler-Aware Integration: The prompt concatenation is unrolled and optimized at compile time to avoid dynamic memory operations.
• Quantization of Prompts: The continuous prompts are quantized (e.g., to INT8) post-training to reduce their storage and memory bandwidth requirements. It is well-suited for task-switching via different stored prompt sets.

Traditional Adapters (inserted small FFNs) are adapted for MCUs by replacing resource-intensive operations:

• ReLU or Hard-Swish replaces GELU activation to avoid expensive transcendental functions.
• Bottleneck Dimension Tuning: The adapter's hidden dimension is minimized (e.g., r=1 or 2) and chosen to align with the MCU's SIMD width.
• Fused Operations: The adapter's down-projection, nonlinearity, and up-projection are compiled into a single, efficient kernel to reduce function call overhead and leverage register-level data reuse.

This hybrid approach combines PEFT with pruning to achieve ultra-compact adapters:

• A standard PEFT method (e.g., LoRA) is first applied.
• The resulting adapter weights (the delta) are then pruned using structured methods (e.g., pruning entire rows/columns).
• The pruned, sparse delta is encoded in an efficient format like CSR (Compressed Sparse Row) for MCU inference.
• This creates a hierarchical efficiency gain: efficient fine-tuning followed by efficient compression, yielding adapters often under 10KB.

The design and selection of PEFT algorithms based on the specific architectural features of the target Microcontroller Unit (MCU) or Neural Processing Unit (NPU). Key considerations include:

• Supported numerical precision (e.g., INT8, FP16)
• Memory hierarchy (SRAM vs. Flash)
• Availability of specialized accelerator cores (DSP, AI) This ensures the fine-tuning process itself is efficient on the target silicon.

A training regimen that simulates the effects of low-precision quantization (e.g., to 8-bit integers) during the fine-tuning of the adapter parameters. This is critical for MCU deployment because:

• It ensures the adapted model remains accurate after quantization.
• It accounts for the non-linear effects of quantization on gradients.
• It produces adapters compatible with MCU inference engines like TensorFlow Lite for Microcontrollers.

A self-contained, resource-constrained software routine that executes on an MCU to perform local model updates via PEFT. It manages the full lifecycle within a strict power and memory budget:

• Local data collection and buffering
• Forward and backward passes through the base model + adapter
• Optimizer step (e.g., SGD) applied only to adapter weights
• Checkpointing the updated adapter to non-volatile memory

A software update strategy where only the small, trained adapter weights (the 'delta') are distributed to edge devices, instead of a full multi-megabyte model. This is essential for MCUs due to:

• Limited bandwidth for Over-the-Air (OTA) updates
• Constrained flash memory for storing multiple model variants
• The ability to hot-swap adapters by loading different deltas into RAM at runtime for task switching.

A decentralized learning paradigm where a fleet of MCUs collaboratively train PEFT adapters on their local sensor data. Only the small adapter updates (e.g., LoRA matrices) are sent to a central server for secure aggregation, not the raw data. This reduces communication costs by orders of magnitude compared to full-model federated learning, making it feasible for low-power wireless MCUs.