MCUNet

TinyNAS employs a two-stage search that first optimizes the search space itself based on the target hardware's resource constraints, then searches for the optimal architecture within that constrained space. This hardware-aware pruning of the search space ensures that candidate architectures are inherently feasible for microcontrollers before any training begins, dramatically reducing the search cost and preventing the selection of models that would exceed SRAM limits during inference.

TinyEngine is a specialized inference runtime that replaces generic deep learning libraries with code-generation-based optimizations tailored to the specific model architecture. It eliminates runtime overhead by:

• In-place depthwise convolution to reduce peak memory
• Operator fusion to minimize intermediate tensor allocations
• Loop unrolling and tiling optimized for Cortex-M cache hierarchies This co-designed library removes the reliance on heavy frameworks like CMSIS-NN for every operation, achieving a 4.1x speedup over TensorFlow Lite Micro on the same hardware.

To handle large input resolutions without exceeding SRAM, MCUNet introduces patch-based inference. The input image is divided into overlapping patches that are processed sequentially, with feature maps stitched together at a later stage. This technique decouples the peak activation memory from the input resolution, enabling models to process 224x224 images on devices with only 256KB of SRAM—a feat previously considered impossible for microcontrollers.

Traditional layer-by-layer execution stores the full activation map of one layer before computing the next, creating a high peak memory watermark. MCUNet's TinyEngine redistributes this computation by fusing early layers with depthwise convolutions and scheduling partial computations. This reduces the peak SRAM footprint by up to 4.5x compared to standard inference engines, allowing deeper networks to fit within the same memory budget.

Unlike interpreter-based runtimes that fetch and execute generic kernel implementations at runtime, TinyEngine uses a compiler-based approach that generates specialized C code for the exact model graph. This eliminates:

• Kernel dispatch overhead
• Unnecessary memory copies between operators
• Runtime shape inference for static graphs The result is deterministic, minimal-latency execution suitable for hard real-time signal processing on bare-metal systems.

MCUNet's core innovation is the co-search of the neural architecture and the memory scheduling strategy. During the search phase, candidate architectures are evaluated not just on accuracy and FLOPs, but on their actual peak SRAM usage under TinyEngine's optimized scheduling. This joint optimization discovers non-intuitive architectural patterns—such as specific kernel sizes and channel counts—that maximize accuracy while aligning perfectly with the compiler's ability to fuse and tile operations.

A critical memory-saving technique where a large input image is processed in overlapping patches. Instead of storing the entire intermediate feature map in SRAM, MCUNet computes one spatial region at a time, drastically reducing the peak activation footprint. This is conceptually similar to tiling in compiler optimization but applied at the neural network architecture level. It trades some recomputation for the ability to run high-resolution inputs on devices with only 256KB of SRAM.

MCUNet's core philosophy is joint optimization of the neural architecture and the inference library. Traditional workflows design a model, then compress it, then write a runtime. MCUNet collapses this into a single feedback loop. The search space is defined by what TinyEngine can execute efficiently, and TinyEngine is specialized for the patterns TinyNAS discovers. This co-search is what enables ImageNet-scale accuracy on off-the-shelf Cortex-M7 processors with 1MB of flash and 320KB of SRAM.

MCUNet integrates INT8 quantization directly into the architecture search process. The accuracy evaluation during NAS is performed on quantized models, not floating-point proxies. This ensures the discovered architecture is inherently robust to quantization error. Combined with batch normalization folding and per-channel symmetric quantization, the final model is deployed as a pure integer graph, maximizing throughput on the CMSIS-NN kernel library.