Hardware-Aware NAS

HW-NAS directly incorporates hardware cost models as primary optimization objectives, moving beyond just accuracy and model size. These models predict key metrics on the target device:

• Latency: Inference time measured in milliseconds, often using lookup tables or lightweight neural predictors.
• Energy Consumption: Estimated in millijoules per inference, critical for battery-powered microcontrollers.
• Peak Memory Usage: The maximum SRAM/Flash footprint during inference.
• Multiply-Accumulate (MAC) Operations: A proxy for computational complexity. By integrating these costs, the search algorithm penalizes architectures that are inefficient on the specific target silicon, such as an Arm Cortex-M4 or an Edge TPU.

Modern HW-NAS often employs Differentiable Architecture Search (DARTS) frameworks extended with a hardware-aware loss term. The search optimizes a bi-level objective: L = L_task(Accuracy) + λ * L_hardware(Cost) Where L_hardware encodes the penalty for high latency or energy use, and λ controls the trade-off. This allows the use of efficient gradient-based optimization to explore the architecture space while simultaneously steering the search toward hardware-efficient sub-networks. The search space itself is typically composed of mobile-optimized building blocks like inverted residual blocks or depthwise separable convolutions.

To avoid the intractable cost of training and profiling every candidate architecture on real hardware, HW-NAS uses efficient proxies:

• Zero-Cost Proxies: Metrics like synaptic flow or gradient norm that correlate with final trained performance and can be computed from an untrained, initialized network in seconds.
• On-Device Measurement: For final candidate evaluation, the search pipeline performs lightweight on-device profiling. This involves compiling the network for the target (e.g., using TensorFlow Lite for Microcontrollers) and executing a forward pass to capture true latency and memory usage, providing ground-truth validation of the cost models.

A pivotal efficiency technique is the use of a weight-sharing supernet or Once-For-All network. A single, over-parameterized network is trained to encompass many sub-networks of varying depths, widths, and kernel sizes. During search, candidate architectures are sampled as different paths through this supernet, inheriting pre-trained weights. This eliminates the need to train each candidate from scratch. The HW-NAS algorithm searches this elastic search space, evaluating the hardware cost of each sub-network to find the optimal configuration for a given microcontroller's memory and latency budget.

Advanced HW-NAS frameworks like MCUNet demonstrate co-design, where the neural architecture and the underlying inference engine are optimized simultaneously.

• Neural Search (TinyNAS): Discovers the model topology.
• System Search (TinyEngine): Optimizes the memory scheduling, loop tiling, and in-place computation of the inference runtime. This tight integration allows the system to fit ImageNet-scale models into extreme constraints (e.g., 256KB of SRAM) by ensuring the generated architecture is not only efficient in theory but also executable by a highly tailored runtime that minimizes memory fragmentation and overhead.

The output of HW-NAS is typically not a single model, but a Pareto-optimal frontier of models representing the best possible trade-offs between accuracy and hardware cost (e.g., latency vs. accuracy). This allows a developer to select the appropriate model for their specific deployment scenario:

• High-Accuracy Point: For a device with more resources.
• Low-Latency Point: For real-time sensor processing.
• Low-Energy Point: For always-on, battery-critical applications. The search algorithm is explicitly designed to explore and populate this frontier, providing actionable options for different product requirements within the same hardware family.

HW-NAS employs automated algorithms to explore a vast space of potential architectures under hardware constraints.

• Reinforcement Learning (RL): An agent (e.g., an RNN controller) proposes architectures, receives a reward based on target hardware performance (latency, energy), and learns to generate better designs.
• Evolutionary Algorithms: A population of candidate architectures undergoes mutation and crossover; the fittest (best accuracy under constraints) are selected for the next generation.
• Differentiable NAS (DARTS): Treats the search space as a continuous super-network. Architecture parameters are learned via gradient descent alongside weights, making search efficient but often requiring proxy metrics.
• One-Shot / Weight-Sharing: A single over-parameterized super-network is trained once. Sub-networks are sampled and evaluated without retraining, drastically reducing search cost. Example: Once-For-All (OFA) network.

A core challenge is predicting how an architecture will perform on real silicon without costly deployment for every candidate.

• Pre-Built Latency/Energy Look-Up Tables (LUTs): Measure the cost of each operator (conv3x3, FC layer) on the target hardware. The total model cost is estimated by summing the costs of its constituent operators.
• On-Device Measurement: Deploys and profiles candidate models directly on the target device (e.g., microcontroller, NPU) for ground-truth metrics. Accurate but slower.
• Analytical Performance Models: Use mathematical models of the hardware's compute and memory hierarchy to predict execution time and energy consumption from the model's computational graph.
• Neural Predictors: Train a small neural network to predict the latency or energy of a candidate architecture given its description, learning complex, non-linear interactions between operators.

The most advanced HW-NAS systems jointly optimize the neural network architecture and the low-level inference software stack.

• MCUNet: A landmark framework that co-designs TinyNAS (for architecture search) and TinyEngine (a memory-aware inference library). It enables ImageNet-scale vision models to run on microcontrollers with under 256KB of SRAM by optimizing dataflow and operator scheduling.
• Once-For-All (OFA): Trains a single large network that supports many sub-networks via weight sharing. Enables instant specialization for different hardware constraints (latency from 10ms to 80ms) without retraining, ideal for heterogeneous edge device fleets.
• ProxylessNAS: Conducts the architecture search directly on the target task (e.g., ImageNet) and hardware, eliminating the need for proxy tasks. It uses path-level binarization and gradient-based methods to directly optimize for metrics like latency on a mobile GPU.

The set of possible architectures the NAS algorithm can explore is critical. For TinyML, spaces are constrained for microcontroller feasibility.

• Cell-Based Search: Searches for a optimal repeating computational "cell" (e.g., Normal Cell, Reduction Cell). The full network is built by stacking discovered cells. Reduces search complexity.
• Macro-Architecture Search: Searches over high-level dimensions: network depth (number of layers), layer width (number of channels), and kernel sizes (3x3, 5x5).
• Operator Choices: At each layer, the search selects from a set of efficient, hardware-friendly operations:
  • Depthwise separable convolution
  • Inverted residual blocks
  • Squeeze-and-excitation blocks
  • Identity (skip connection)
• Constraint-Aware Pruning: The search space itself can be dynamically pruned to exclude architectures that obviously violate hard memory or latency budgets.

HW-NAS must balance the primary objective (accuracy) with multiple, often competing, hardware constraints.

• Pareto Frontier: The goal is to find architectures that are "Pareto optimal"—where you cannot improve one metric (e.g., accuracy) without worsening another (e.g., latency or energy).
• Weighted Sum Method: Combines objectives into a single scalar cost function: Cost = Loss + λ * Latency. The hyperparameter λ controls the trade-off.
• Constraint-Based Optimization: Formulates hardware metrics as hard constraints (e.g., Latency < 10ms, Peak RAM < 200KB). The search maximizes accuracy only within the feasible region.
• Evolutionary algorithms like NSGA-II are commonly used to directly discover a diverse set of models along the Pareto frontier in a single search run.

Neural Architecture Search (NAS) is the overarching automated process for discovering optimal neural network topologies. Unlike manual design, NAS uses algorithms—such as reinforcement learning, evolutionary strategies, or gradient-based methods—to explore a vast search space of possible layer types, connections, and hyperparameters. Hardware-Aware NAS is a specialized variant where the search objective includes direct hardware performance metrics like latency or energy, rather than just accuracy.

A Once-For-All (OFA) network is a weight-sharing supernet trained to contain a massive number of sub-networks of varying depths, widths, and kernel sizes. This enables efficient hardware-aware NAS by allowing the search algorithm to evaluate the performance of many different candidate architectures on target hardware without retraining each one from scratch. It decouples training from search, making it practical to find specialized models for diverse microcontroller constraints.

ProxylessNAS is a direct hardware-aware NAS method that searches for architectures on the actual target task and hardware, eliminating the need for a proxy (like a smaller dataset). It uses techniques like path-level binarization and gradient-based optimization to incorporate real hardware feedback—such as measured latency—directly into the search loop. This approach avoids the inaccuracies that can arise from proxy metrics, leading to models that are precisely tuned for a specific device's performance profile.

MCUNet represents the state-of-the-art in hardware-aware co-design for microcontrollers. It consists of two tightly integrated components:

• TinyNAS: A hardware-aware NAS algorithm that searches for networks within extremely tight memory budgets (e.g., < 256KB SRAM).
• TinyEngine: An inference library that generates highly optimized, memory-aware code for the discovered architecture. This co-design ensures the neural architecture and the underlying inference engine are jointly optimized, enabling ImageNet-scale vision tasks on commodity MCUs.

Differentiable Architecture Search (DARTS) is a gradient-based NAS method that relaxes the discrete search space into a continuous one. It treats the choice between operations (e.g., 3x3 conv, 5x5 conv, skip connect) as a set of continuous architecture parameters optimized alongside network weights via standard gradient descent. While not inherently hardware-aware, DARTS provides the efficient, differentiable search foundation that many subsequent hardware-aware NAS methods build upon by incorporating hardware cost as a differentiable term in the loss function.

EfficientNet introduced a compound scaling method that uniformly scales network depth, width, and resolution using a fixed set of coefficients. While not a search algorithm itself, this principled scaling law is a critical benchmark and inspiration for hardware-aware NAS. Variants like EfficientNet-Lite are manually derived for edge hardware, but HW-NAS automates the discovery of such optimally scaled and factorized architectures under specific computational budgets, often outperforming fixed scaling rules.