MCUNet

TinyNAS is the neural architecture search (NAS) component of MCUNet. It automatically designs efficient convolutional neural networks (CNNs) under the extreme memory constraints of microcontrollers (e.g., < 512KB of SRAM). Unlike cloud-based NAS, TinyNAS directly optimizes for peak memory usage and latency on the target hardware, using a two-stage search process: first in a large design space for accuracy, then a specialized search for memory and latency. This results in models that are not just small, but are architecturally tailored to fit within a device's specific SRAM and Flash limits.

TinyEngine is the inference runtime engine co-designed with TinyNAS. It is a memory-aware and lightweight library that replaces generic frameworks like TensorFlow Lite for Microcontrollers. Its key innovations include:

• In-place Depth-wise Convolution: Drastically reduces peak memory by performing computations directly on the input buffer.
• Operator Fusion with Static Memory Planning: Fuses layers (e.g., convolution, batch norm, ReLU) and pre-allocates all tensor buffers at compile time, eliminating dynamic memory allocation overhead and fragmentation.
• Int8 Kernel Support: Provides highly optimized kernels for quantized models. TinyEngine's scheduling ensures the model designed by TinyNAS executes within the planned memory budget.

The breakthrough of MCUNet is the joint optimization of TinyNAS and TinyEngine. Traditional flow designs a model first, then tries to fit it onto hardware, often failing. MCUNet's co-design treats the model architecture and inference scheduling as one optimization problem. TinyNAS searches for models that are efficient given TinyEngine's memory-saving execution patterns, and TinyEngine is optimized to run the types of layers TinyNAS frequently selects. This closed-loop optimization is why MCUNet can achieve high accuracy (e.g., >70% on ImageNet) on an Arm Cortex-M7 chip with only 320KB SRAM, where prior methods failed.

MCUNet fully embraces post-training quantization and integer arithmetic to maximize efficiency. The framework is designed to work seamlessly with 8-bit (INT8) quantized models, which reduce model size by 4x compared to FP32 and enable faster computation on microcontrollers lacking Floating-Point Units (FPUs). TinyEngine includes hand-optimized INT8 kernels for core operations. The NAS process in TinyNAS can also account for quantization-aware training, ensuring the discovered architectures are robust to the precision loss, maintaining accuracy while enabling milliwatt-scale inference.

TinyNAS does not search over arbitrary architectures. Its search space is carefully constrained by hardware-level insights. This includes favoring:

• Depth-wise separable convolutions over standard convolutions for better compute/memory trade-off.
• Specific kernel sizes (e.g., 3x3, 5x5) and expansion ratios that align well with efficient memory access patterns.
• Layer structures that enable TinyEngine's in-place computation optimizations. This hardware-aware search space pruning makes the NAS process faster and more likely to yield practically deployable models, directly targeting metrics like inferences per joule (IPJ).

MCUNet refers to a series of frameworks and model families. Key versions include:

• MCUNet (v1): Introduced the core co-design of TinyNAS and TinyEngine, achieving ImageNet classification on 320KB SRAM.
• MCUNet (v2): Extended the approach to vision transformers (ViTs), co-designing TinyViT models and a TinyEngine optimized for transformer attention mechanisms on microcontrollers.
• MCUNet (v3): Focused on on-device training (fine-tuning), introducing TinyTrainingEngine to enable adaptation with <256KB memory. Each evolution addresses new challenges, expanding the frontier of what's possible with TinyML on microcontroller units (MCUs).

MCUNet's performance is rigorously measured on commercial off-the-shelf microcontrollers, defining the state of the art for TinyML.

• Arm Cortex-M7 (STM32F746): 216 MHz, 320KB SRAM. MCUNet runs ImageNet-scale inference in under 500ms.
• Arm Cortex-M4 (nRF52840): 64 MHz, 256KB SRAM. Runs keyword spotting in ~30ms, enabling real-time audio processing.
• Memory Breakdown: For a VWW model, weights may occupy ~260KB Flash, with activations consuming ~60KB SRAM, fitting entirely in on-chip memory to avoid slow external RAM.
• Metric: Achieves thousands of Inferences Per Joule (IPJ), enabling year-long battery life for periodic sensing applications.

The broader methodology underpinning MCUNet's TinyNAS. HW-NAS automates model design while directly incorporating hardware performance as a constraint or objective. For MCUNet, the search space and cost models are specifically tailored for ARM Cortex-M series processors, evaluating candidates based on their estimated latency, SRAM usage, and Flash storage requirements. This ensures the final architecture is not just accurate, but practically deployable.

The total amount of RAM (SRAM) and Flash (ROM) memory required to store and execute a model. This is the primary constraint in TinyML. MCUNet explicitly breaks down and optimizes for both:

• Peak SRAM Usage: The maximum working memory needed during inference. TinyEngine minimizes this via in-place operations.
• Flash Footprint: The storage space for the model's weights and code. TinyNAS designs compact architectures, and quantization (e.g., to 8-bit integers) further reduces this. Successful deployment requires the footprint to fit within the target MCU's specs, often as little as 256KB of Flash and 32KB of SRAM.

The dominant family of 32-bit microcontroller cores that are the primary target for MCUNet and TinyML. Key characteristics include:

• Ultra-low power consumption (milliwatt range).
• Limited memory (often < 1MB Flash, < 512KB SRAM).
• Lack of an OS or a simple RTOS.
• May lack a Floating-Point Unit (FPU), necessitating fixed-point or integer arithmetic. MCUNet is specifically engineered to unlock complex computer vision and audio models on these ubiquitous, constrained devices, such as the Cortex-M7 or Cortex-M33.