Hardware-Aware Neural Architecture Search

The search space is defined not only by architectural operations (e.g., convolution types, attention blocks) but also by hardware-compatible configurations. This includes:

• Operator-level choices (e.g., depthwise vs. standard convolution).
• Quantization-aware blocks designed for efficient INT8 execution.
• Sparsity patterns (e.g., structured N:M) that align with accelerator capabilities. The search algorithm evaluates each candidate architecture against a Pareto frontier balancing accuracy, latency, and memory use.

Instead of relying on theoretical FLOPs or parameter counts, hardware-aware NAS uses precise measurements from the target device or a high-fidelity simulator.

• Real hardware profiling: The search process executes candidate models (or subgraphs) on the actual microcontroller (MCU) or development board to measure true latency and peak memory usage.
• Learned latency predictors: To avoid the cost of profiling every candidate, a surrogate model (e.g., a neural network or lookup table) is trained to predict latency based on architectural features and hardware specifications.
• Memory bottleneck modeling: Profiles SRAM/Flash access patterns and contention, which are often the limiting factor on MCUs.

Modern hardware-aware NAS often employs Differentiable Architecture Search (DARTS)-like methods, where the search is continuous and optimized via gradient descent. A hardware cost term is directly integrated into the loss function: Total Loss = Task Loss (e.g., Cross-Entropy) + λ * Hardware Loss The Hardware Loss can penalize predicted latency, energy consumption, or model size. The weighting factor (λ) controls the trade-off between accuracy and efficiency. This allows the search to smoothly navigate towards architectures that are optimal for the specific silicon constraints.

The search process accounts for optimizations performed by the target inference compiler or runtime (e.g., TensorFlow Lite for Microcontrollers, Apache TVM).

• Kernel fusion: Favors operator sequences that the compiler can fuse into a single, efficient kernel to reduce overhead.
• Data layout: Considers memory alignment and tensor formats (NHWC vs. NCHW) preferred by the underlying hardware libraries.
• Supported ops: Constrains the search to use only operators that have highly optimized implementations for the target MCU or Neural Processing Unit (NPU).

To manage the extreme cost of searching for each new hardware target, a Once-For-All (OFA) approach is used. A single, large supernet is trained containing many possible subnetworks.

• The supernet is trained with techniques like weight sharing and progressive shrinking.
• After training, specialized subnetworks for different hardware profiles (e.g., a 100KB model for Device A, a 50KB model for Device B) can be extracted without retraining.
• The hardware-aware search then becomes a fast process of evaluating and selecting the best subnetwork from the supernet for a given device's latency and memory budget.

A core outcome of hardware-aware NAS is a Pareto-optimal set of models, where no model can be improved in one metric (e.g., latency) without worsening another (e.g., accuracy). This set is specific to each hardware platform.

• A model optimal for a GPU may be inefficient on an ARM Cortex-M MCU due to different memory hierarchies and compute units.
• The process highlights that the best neural architecture is inherently hardware-dependent. This leads to families of models like MobileNetV3 (for mobile CPUs) or MCUNet (for microcontrollers), each born from hardware-aware search spaces.

Neural Architecture Search is the foundational automated process for discovering high-performing neural network topologies. It defines a search space of possible operations (e.g., convolution types, activation functions) and connections, uses a search strategy (e.g., reinforcement learning, evolutionary algorithms) to explore it, and a performance estimator (like a validation accuracy predictor) to evaluate candidates. Hardware-aware NAS extends this core by incorporating device-specific metrics into the estimator.

A Once-For-All network is a trainable supernet containing a vast number of possible subnetworks within a single weight set. It is trained once via progressive shrinking and then can instantly provide numerous specialized, efficient submodels for different latency, memory, or power constraints without retraining. This is a pivotal enabler for hardware-aware NAS, allowing rapid evaluation of candidate architectures on a target device's performance profile.

Model compression is the overarching discipline of reducing a neural network's computational and memory footprint. Key techniques include:

• Quantization: Reducing numerical precision of weights/activations.
• Pruning: Removing redundant parameters.
• Knowledge Distillation: Training a small model to mimic a large one. Hardware-aware NAS often co-designs the architecture with these techniques, searching for networks that are inherently efficient and amenable to further compression.

These are hand-designed network topologies optimized for severe microcontroller constraints, serving as benchmarks and inspiration for NAS. Key design principles include:

• Depthwise separable convolutions (e.g., MobileNet) to reduce parameters.
• Bottleneck layers (e.g., SqueezeNet) to limit feature map channels.
• Micro-architecture choices like kernel size, activation functions, and skip connections. Hardware-aware NAS automates the exploration of these principles within a defined search space tailored for edge devices.

Software toolchains that enable the end-to-end development and deployment of microcontroller ML, providing the essential infrastructure for hardware-aware NAS. They offer:

• Hardware-in-the-loop profiling to measure real latency, memory, and energy.
• Model conversion & compilation (e.g., to TensorFlow Lite for Microcontrollers).
• Deployment pipelines for testing on actual silicon. Frameworks like TensorFlow Lite Micro, Apache TVM, and MCUNet provide the profiling data and deployment targets that guide the hardware-aware search process.

The core technical challenge in hardware-aware NAS is predicting a candidate architecture's performance on target hardware without exhaustive training and deployment. Strategies include:

• Zero-cost proxies: Using simple metrics like synaptic flow.
• Learned predictors: Training a surrogate model (e.g., a neural network) on architecture-performance pairs.
• Hardware-in-the-loop measurement: Directly profiling a subset of candidates on the device to calibrate the predictor. The efficiency and accuracy of this estimator directly determine the practicality of the NAS process.