Hardware-Aware Neural Architecture Search (HW-NAS)

HW-NAS treats hardware metrics as primary objectives within the search process, creating a Pareto frontier of optimal trade-offs. The search algorithm must balance:

• Model Accuracy: The primary predictive performance metric.
• Hardware Efficiency: Metrics like inference latency, energy consumption (Inferences Per Joule), and peak memory usage.
• Model Size: Parameters and activations impacting memory footprint.

Unlike standard NAS, which often prioritizes accuracy alone, HW-NAS explicitly penalizes architectures that are inefficient on the target hardware, such as a mobile CPU, microcontroller, or NPU like the Arm Ethos-U55.

A defining feature of HW-NAS is the direct, iterative measurement of candidate architectures on real or emulated target hardware. This moves beyond theoretical estimates (e.g., FLOPs count) to capture real-world bottlenecks.

Profiling Methods Include:

• On-device Measurement: Deploying and timing each candidate model on the physical hardware (slow but accurate).
• Performance Predictors: Training a surrogate model (e.g., a neural network) to predict latency/energy based on architectural features, which is then used to cheaply score new candidates.
• Emulation & Simulation: Using cycle-accurate simulators or hardware performance models (e.g., Gem5, Timeloop) for early-stage estimation.

This ensures the final model is optimized for the specific memory hierarchy, cache behavior, and supported operators of the deployment platform.

The search space—the set of all possible neural network architectures the algorithm can explore—is explicitly constrained by hardware realities from the outset. This involves:

• Operator Selection: Including only hardware-friendly operations (e.g., depthwise convolutions over standard convolutions, fixed-point arithmetic-compatible activations).
• Kernel Size & Channel Constraints: Limiting dimensions to avoid memory blow-ups and align with accelerator tile sizes.
• Macro-Architecture Templates: Using proven, efficient backbones like MobileNetV3 or MCUNet as starting scaffolds.
• Support for Compression Primitives: Designing spaces that inherently include quantization-aware blocks or prunable structures.

This constrained design prevents the search from wasting time on architectures that are fundamentally incompatible with the milliwatt computing budget of a microcontroller.

Modern HW-NAS often employs efficient differentiable or one-shot search strategies to manage the computational cost of evaluating thousands of architectures.

• Differentiable Architecture Search (DARTS): Treats the architecture selection as a continuous optimization problem. A supernet containing all candidate operations is trained; the search learns a distribution over operations via gradient descent, later discretizing to the final hardware-optimal sub-network.
• One-Shot / Weight-Sharing: A single, over-parameterized supernet (e.g., a Once-for-All Network) is trained once. The search then evaluates sub-networks by sharing the supernet's weights, avoiding individual training. A predictor ranks sub-networks based on their hardware performance.
• Evolutionary & Reinforcement Learning: These remain common, where a controller (RL agent) or population (evolution) proposes architectures, which are then scored by the hardware performance predictor.

The most advanced HW-NAS frameworks practice full-stack co-design, where the neural architecture and the underlying inference engine (like TensorFlow Lite for Microcontrollers or a custom TinyEngine) are optimized jointly.

Key Co-Design Optimizations:

• Kernel-Aware Search: The search algorithm is aware of which operations are highly optimized in the target inference library (e.g., CMSIS-NN kernels for Arm Cortex-M).
• Memory Scheduling: Architectures are chosen to minimize intermediate tensor buffering and leverage static memory allocation patterns.
• Compiler Optimizations: The search considers the benefits of operator fusion performed by the compiler, favoring sequences of operations that can be fused into a single, efficient kernel.

This tight integration, exemplified by frameworks like MCUNet, yields models that are not just theoretically efficient but are perfectly tailored to the runtime's execution model.

The output of HW-NAS is not a single "best" model, but a family of models representing the best possible accuracy for a given hardware constraint. This enables flexible deployment across a heterogeneous device fleet.

Practical Outcomes:

• A developer can select a model from the Pareto frontier that meets a specific latency SLA (e.g., < 100ms) or memory budget (e.g., < 256KB RAM).
• Enables dynamic scaling where a device selects a larger, more accurate model when plugged in, and a tiny, efficient model when on battery power.
• Provides clear trade-off analysis for product managers: "A 2% accuracy drop yields a 50% reduction in energy use."

This characteristic directly addresses the core challenge of TinyML: achieving usable performance under severe, non-negotiable physical constraints.

HW-NAS designs ultra-efficient convolutional neural networks (CNNs) for battery-powered smart cameras and doorbells. The goal is to run a simple visual wake word or anomaly detection model continuously at milliwatt power levels.

• Constraint: Strict inference latency (<100ms) and power budget for real-time response.
• HW-NAS Role: Searches for models that balance accuracy with the energy consumption profile of the device's Vision Processing Unit (VPU) or microNPU.
• Outcome: Models that can distinguish 'person' vs. 'no person' for years on a battery, triggering higher-power systems only when needed.

Designing audio models for voice-controlled IoT devices (e.g., light switches, appliances). HW-NAS optimizes for the Digital Signal Processor (DSP) blocks and memory hierarchy of low-power ARM Cortex-M microcontrollers.

• Challenge: Fit a model detecting 10-20 keywords into < 100KB of RAM and Flash.
• HW-NAS Optimization: Searches architectures using depthwise separable convolutions and GRU/LSTM cells optimized for the MCU's fixed-point arithmetic units.
• Result: High-accuracy, sub-50ms latency models that enable responsive, private, on-device voice interfaces.

Neural Architecture Search (NAS) is the automated, overarching process for designing optimal neural network architectures. It uses methods like reinforcement learning, evolutionary algorithms, or gradient-based optimization to explore a vast search space of possible layer types, connections, and hyperparameters. HW-NAS is a specialized variant where the search objectives or constraints are directly tied to hardware performance metrics.

• Core Goal: Automate model design to maximize predictive accuracy.
• Search Space: Defines the set of all possible architectures (e.g., choice of operations, number of layers, filter sizes).
• Search Strategy: The algorithm used to navigate the search space efficiently.
• Performance Estimation: How the accuracy of a candidate architecture is evaluated, often via proxy (e.g., training on a subset of data).

A Once-for-All (OFA) network is a training and deployment paradigm central to efficient HW-NAS. Instead of searching for and training a single model per hardware target, a single large supernet is trained. This supernet contains a massive number of smaller sub-networks within its weight-sharing structure. After training, specialized search algorithms can rapidly extract optimal sub-networks tailored for different hardware constraints (e.g., 50ms latency vs. 200ms, 256KB vs. 1MB memory) without any retraining.

• Key Benefit: Eliminates the need for costly re-searching and retraining for each new device profile.
• Elastic Dimensions: The supernet is designed with variable depth, width, and kernel sizes.
• Deployment: Enables a 'train once, deploy anywhere' workflow for heterogeneous edge hardware.

Model compression is a suite of techniques used to reduce the computational and memory footprint of neural networks, making them viable for edge deployment. While HW-NAS designs efficient models from scratch, compression techniques optimize existing models. These techniques are often complementary and can be integrated into the NAS search space.

• Quantization: Reduces numerical precision of weights/activations (e.g., FP32 to INT8). Directly lowers memory footprint and accelerates computation on integer units.
• Pruning: Removes redundant weights or neurons, creating a sparse model. Reduces model size and FLOPs.
• Knowledge Distillation: Trains a small student model to mimic a larger teacher model, preserving accuracy in a compact form.
• Relationship to HW-NAS: Compression metrics (e.g., bit-width, sparsity ratio) can be explicit search dimensions in an HW-NAS algorithm.

Inference latency and memory footprint are the primary hardware metrics optimized by HW-NAS. They are non-differentiable, device-specific constraints that must be accurately profiled during the search.

• Inference Latency: The time from input presentation to output generation. HW-NAS uses latency predictors or on-device look-up tables to estimate this for each candidate architecture without full deployment.
• Memory Footprint: The total RAM (for activations) and Flash/ROM (for model weights) required. This is a hard constraint on microcontrollers (often 256KB-1MB total).
• Worst-Case Execution Time (WCET): For real-time systems, the maximum possible latency is more critical than average latency. HW-NAS for safety-critical applications must consider WCET.
• Milliwatt Computing: Ultimately, latency and memory translate into energy consumption. The goal is to maximize Inferences Per Joule (IPJ).

TinyML co-design represents the most advanced application of HW-NAS principles, where the neural network architecture and the underlying inference engine are optimized jointly. MCUNet is a seminal example, featuring TinyNAS for hardware-aware search and TinyEngine for memory-optimized execution.

• TinyNAS: An HW-NAS method that operates under the extreme memory constraints of microcontrollers (e.g., < 512KB). It searches for models that fit within a device's SRAM to avoid slow flash memory access.
• TinyEngine: A dedicated inference library that uses static memory allocation and operator fusion to minimize memory overhead. It is generated specifically for the model found by TinyNAS.
• Result: This co-design pushes the Pareto frontier of accuracy versus resource use further than optimizing either the model or the engine alone.

Hardware accelerators like the Arm Ethos-U55 microNPU are specialized processors that dramatically improve the efficiency of ML workloads. HW-NAS must be explicitly accelerator-aware to leverage these units. The search space and cost models must incorporate the accelerator's unique capabilities and limitations.

• Accelerator-Aware Search: The NAS algorithm evaluates candidates based on how well they map to the accelerator's compute fabric, supported operations (ops), and memory hierarchy.
• Compiler Toolchain: The output of HW-NAS must be compatible with the accelerator's compiler (e.g., Vela compiler for Ethos). The compiler performs further low-level optimizations like scheduling and tiling.
• Digital Signal Processing (DSP) Blocks: Many microcontrollers have DSP blocks for efficient multiply-accumulate (MAC) operations. HW-NAS can favor operations that map cleanly to these blocks.
• Heterogeneous Targets: Advanced HW-NAS frameworks can search for models that run across multiple compute units (CPU, DSP, NPU) on a single System-on-Chip (SoC).