Microcontroller NAS (MCU-NAS)

MCU-NAS enforces strict peak RAM (activation memory) and flash (model storage) constraints, often in the range of tens to hundreds of kilobytes. The search algorithm must discover architectures where both the weights and the intermediate tensors during inference fit entirely within on-chip SRAM to avoid costly external memory access.

• Model Size: Optimizes for a minimal number of parameters to fit within limited flash memory.
• Activation Memory: Considers the memory footprint of feature maps during inference, which is often the limiting factor for network depth and width.
• Example: A search might be constrained to < 256KB of total memory, forcing the use of efficient operations like depthwise separable convolutions and limiting channel counts.

The search objective incorporates a power or energy cost model to minimize inference energy, which is critical for battery-powered IoT devices. This goes beyond just reducing operations; it considers the actual energy profile of different operations on the target MCU.

• Operation-Level Cost: Differentiates between the energy cost of a standard convolution, depthwise convolution, and fully connected layer on the target hardware.
• System Power States: May co-optimize for architectures that allow the MCU to return to a deep sleep state quickly.
• Goal: Discovers models that complete inference within a strict energy budget (e.g., < 100 µJ per inference), enabling years of operation on a coin-cell battery.

MCU-NAS targets real-time performance on single-core, low-clock-speed CPUs (e.g., Arm Cortex-M series at 80-200 MHz). The search uses a latency constraint or cost model to ensure inference completes within a required timeframe, often tens to hundreds of milliseconds.

• Clock-Cycle Awareness: Estimates latency based on the number and type of operations, considering the lack of parallel processing units like GPUs.
• Memory Hierarchy Effects: Accounts for slowdowns due to cache misses or accesses to slower flash memory.
• Contrast with Mobile NAS: While mobile NAS (e.g., for smartphones) optimizes for multi-core GPUs, MCU-NAS assumes a simple, sequential execution environment.

MCU-NAS typically assumes deployment using integer-only inference (e.g., 8-bit or 16-bit precision). The search process either discovers architectures that are natively robust to quantization or integrates quantization-aware training (QAT) simulation during the search.

• Quantization-Aware Search (QA-NAS): Evaluates candidate architectures using simulated integer arithmetic to ensure accuracy is maintained post-quantization.
• Precision Search: Can include the choice of per-layer bit-width (e.g., 8-bit for weights, 16-bit for accumulators) as part of the search space.
• Avoids Floating-Point: Actively discourages operations that require floating-point units (FPUs), which are absent or energy-inefficient on most microcontrollers.

The search space is pruned to only include neural network operators that have efficient implementations on the target MCU. This excludes computationally expensive layers common in cloud models.

• Common Allowed Ops: Depthwise separable convolutions, pointwise convolutions, average pooling, and fully connected layers.
• Excluded Ops: Standard large-kernel convolutions, batch normalization (folded into preceding conv layer at deployment), and complex activations like SiLU.
• Kernel Library Alignment: The search is aligned with the optimized neural network kernels available in frameworks like TensorFlow Lite for Microcontrollers or CMSIS-NN.

MCU-NAS must avoid architectures that are incompatible with the target hardware's microarchitecture. This involves constraints that are not purely about size or speed, but about functional feasibility.

• Memory Alignment: Ensures tensor dimensions are aligned to the MCU's memory bus width to avoid performance cliffs.
• Kernel Size Restrictions: May avoid certain kernel sizes (e.g., 7x7) that are poorly supported by the underlying DSP instructions.
• Avoiding Dynamic Operations: Excludes operations with dynamic memory allocation or control flow, which are unpredictable on bare-metal systems.

This technique directly measures the true latency, peak RAM usage, and energy consumption of candidate architectures by deploying and profiling them on the actual target microcontroller during the search. This replaces inaccurate proxy models with ground-truth hardware feedback, which is critical for MCUs where cache effects and memory bus contention can drastically alter performance predictions.

• Eliminates Proxy Error: Provides exact measurements for flash, SRAM, and current draw.
• Integrates with Search Loop: The NAS controller uses real hardware metrics as the reward signal or constraint.
• Example: A search might reject a candidate that exceeds 64KB of SRAM after on-device measurement, even if a software simulator predicted 60KB.

Given the severe constraints, the NAS search space is aggressively pruned to exclude architectures that are fundamentally incompatible with microcontroller hardware. This is based on hardware-informed heuristics.

• Pruning Rules: Eliminate operations with high latency (e.g., large dense layers), architectures requiring dynamic memory allocation, or topologies that prevent efficient layer fusion.
• Fixed-Point-First Design: The search space may be limited to operators known to be stable with 8-bit or 4-bit integer quantization.
• Benefit: Drastically reduces the number of invalid candidates, making the search tractable on limited compute resources.

Adapts the One-Shot NAS paradigm with a critical focus on memory footprint. A single, weight-shared supernet is trained, but its training and the sampling of sub-networks are constrained by peak memory usage.

• Memory Cost Modeling: A lightweight model or lookup table predicts the SRAM footprint of any sub-graph within the supernet.
• Constrained Weight Sharing: During supernet training, gradient updates are performed only on sub-networks that satisfy a memory budget, ensuring the shared weights are optimized for feasible architectures.
• Outcome: Enables the evaluation of thousands of sub-architectures by simply querying the trained supernet and its memory model, without retraining.

MCU-NAS typically optimizes for three or more competing objectives, seeking architectures on the Pareto frontier where no metric can be improved without degrading another. Common objectives include:

• Model Accuracy (e.g., classification F1-score)
• Peak SRAM Usage (must be under budget, e.g., < 32KB)
• Inference Latency (meets real-time requirement, e.g., < 100ms)
• Energy per Inference (measured in microjoules)

Search strategies like NSGA-II (an evolutionary algorithm) are used to evolve a population of architectures toward this optimal trade-off surface, giving developers a menu of viable models for their specific balance of constraints.

Integrates quantization simulation directly into the architecture search loop. Candidate networks are trained and evaluated with simulated low-precision arithmetic (e.g., INT8), ensuring the discovered architecture is inherently robust to the precision loss of deployment.

• Simulated Quantization: During supernet training or candidate evaluation, weights and activations are quantized and de-quantized in the forward pass.
• Searches for Quantization-Friendly Structures: The algorithm naturally favors operations and activation functions that exhibit low quantization error.
• Result: The final discovered architecture requires minimal or no fine-tuning after post-training quantization, streamlining the deployment pipeline.

The core of the search space defines a set of microcontroller-optimized primitive operations. The NAS algorithm selects and combines these. Common operators include:

• Depthwise Separable Convolutions: Drastically reduces parameters and compute vs. standard convolutions.
• Micro-Fully Connected Layers: Very narrow dense layers with limited input/output features.
• Efficient Activations: Functions like ReLU6 or hard-swish that are cheap to compute on fixed-point units.
• Skip Connections & Add Operations: Used sparingly due to memory overhead of storing residuals.
• Pooling Layers: Fixed, low-overhead operations like average pooling.

The search dynamically constructs a network from this palette, optimizing for the specific sensor data pattern (e.g., 1D time-series vs. 2D audio spectrograms).

The broader category of automated neural network design that incorporates hardware-specific constraints—such as latency, memory footprint, and energy consumption—directly into the search objective. MCU-NAS is a specialized sub-field targeting microcontroller-class devices.

• Key Constraint: Moves beyond just model accuracy to include real-world deployment metrics.
• Search Objective: Typically a weighted sum or multi-objective optimization (e.g., accuracy * (1/latency)).
• Example: Searching for a model that achieves >95% accuracy while using less than 50KB of RAM.

A predictive function used within NAS to estimate the inference latency, peak memory usage, or energy cost of a candidate neural network architecture without deploying it on physical hardware.

• Types: Can be a simple lookup table of layer-wise costs, a linear regression model, or a small neural network predictor.
• Purpose: Drastically accelerates the search by replacing slow, hardware-in-the-loop profiling for most candidates.
• MCU-NAS Challenge: Must be extremely accurate for kilobyte-scale memory and microjoule-level energy predictions, where small errors are significant.

A strict upper bound on the RAM (activation memory) and often flash (model weight) memory a candidate neural network can consume. This is the primary driver in MCU-NAS, where budgets can be as low as 32-256KB of total SRAM.

• Activation Memory: The temporary memory required to store intermediate layer outputs during inference; often the limiting factor.
• Weight Memory: The storage for the model's parameters; targeted via compression techniques like quantization.
• Enforcement: The search algorithm penalizes or discards architectures that violate the memory budget, guiding the search toward feasible designs.

A pioneering one-shot NAS and training paradigm highly relevant for edge deployment. It trains a single, large supernet from which many sub-networks of different depths, widths, and kernel sizes can be extracted without retraining.

• Efficiency for MCUs: After the one-time supernet training, specialized models for different microcontroller memory budgets can be instantiated in seconds.
• Elastic Dimensions: Allows dynamic adjustment of the network to fit a specific device's constraints.
• Use Case: Enables a single model family to serve a heterogeneous fleet of devices, from powerful MPUs to severely constrained MCUs.

A hardware-aware search method that discovers neural network architectures that are inherently robust to quantization. The search is conducted with simulated quantization (e.g., 8-bit integer) in the loop.

• Goal: Find architectures where the accuracy drop from float32 to int8 quantization is minimized.
• MCU-NAS Relevance: Quantization is non-negotiable for MCU deployment to reduce model size and enable fixed-point arithmetic. QA-NAS ensures the discovered architecture won't fail after this essential compression step.
• Process: The search optimizes the quantized accuracy, not the full-precision accuracy.

The integrated philosophy of jointly optimizing neural network architectures and the features of hardware accelerators. While MCU-NAS typically takes fixed hardware as a constraint, co-design explores a broader solution space.

• MCU-NAS as a Subset: MCU-NAS is often the algorithmic component of a co-design flow for a new microcontroller or NPU.
• Feedback Loop: NAS discovers optimal models for a hardware template; hardware design is then tweaked to better support common operations from those models.
• Outcome: Produces highly efficient, locked-down systems like Google's Edge TPU with its supporting model zoo.