Hardware-in-the-Loop Search

The defining characteristic of HITL Search is the direct, in-loop measurement of candidate architectures on the physical target hardware. Instead of relying on proxy metrics or simulated cost models, the search algorithm queries the real device for latency, peak memory usage, and energy consumption. This eliminates the simulation gap—the discrepancy between predicted and actual on-device performance—which is critical for microcontrollers where cache effects, memory bandwidth, and fixed-point arithmetic behavior are difficult to model accurately.

HITL Search operates as a closed-loop control system. The search algorithm (e.g., a controller network or evolutionary optimizer) proposes a candidate architecture. This architecture is compiled, deployed to the target MCU, and executed with a representative input workload. The measured hardware metrics are fed back as a reward signal or constraint penalty to guide the next search iteration. This creates a tight feedback cycle where the search directly optimizes for the empirical performance of the final deployed system.

This methodology is essential for TinyML and microcontroller NAS (MCU-NAS), where constraints are severe and non-negotiable. HITL Search directly enforces:

• Memory Budgets: Ensures total model size (weights + activations) fits within limited SRAM/Flash (e.g., < 512KB).
• Latency Deadlines: Measures inference time to meet real-time sensor processing rates (e.g., < 10ms).
• Energy Caps: Profiles current draw to stay within a device's microjoule-per-inference budget, crucial for battery-powered operation.

HITL Search is often combined with One-Shot NAS and weight sharing to be computationally feasible. A supernet encompassing all candidate operations is trained once. During search, sub-networks are sampled from the supernet and profiled on-device without retraining. This decouples training cost from search cost, allowing thousands of architectures to be evaluated in a HITL loop by simply measuring their execution time and memory footprint on the hardware, which is relatively fast compared to training.

The search inherently optimizes for a Pareto frontier of optimal trade-offs. Objectives typically include:

• Task Accuracy (e.g., classification F1-score)
• Inference Latency (measured on-device in milliseconds)
• Model Memory Footprint (measured in kilobytes)
• Energy per Inference (measured in microjoules) The output is not a single model, but a set of architectures where no metric can be improved without degrading another, allowing engineers to select the best fit for their specific hardware budget.

A practical HITL Search system requires deep integration with the embedded toolchain. This includes:

• Automated Compilation Pipelines: Converting candidate architectures to optimized code (e.g., via TVM, TensorFlow Lite for Microcontrollers).
• On-Device Profilers: Lightweight firmware that runs the model and reports performance counters back to the search host.
• Hardware Abstraction: Managing connections to physical or networked development boards (like STM32 or ESP32 kits) for automated flashing and measurement. This turns the search into a continuous integration process for neural architecture design.

The quintessential use case for Hardware-in-the-Loop Search is discovering neural networks that fit within the severe constraints of microcontrollers (MCUs). These devices have kilobyte-scale SRAM, megabyte-scale flash, and microampere power budgets. The search directly measures:

• Peak RAM usage during inference to avoid crashes.
• Flash memory footprint for model storage.
• Inference latency on the device's CPU, often without a dedicated NPU.
• Energy consumption per prediction, critical for battery-powered IoT sensors. By profiling on hardware like the Arm Cortex-M series or ESP32, the search ensures the final model is deployable, not just theoretically efficient.

This search method is critical for maximizing efficiency on specialized AI accelerators and Neural Processing Units (NPUs) found in smartphones, drones, and cameras. These accelerators have unique memory hierarchies, supported operation sets, and parallel execution units. Hardware-in-the-Loop profiling captures:

• Kernel latency on proprietary systolic arrays or tensor cores.
• Data movement costs between different memory levels.
• Compiler-induced overhead from frameworks like TensorFlow Lite for Microcontrollers or proprietary SDKs. The search can discover architectures that align with the accelerator's strengths, such as preferring depthwise separable convolutions or specific activation functions that are hardware-optimized.

Enterprises often need a single model architecture that performs efficiently across a heterogeneous fleet of devices (e.g., high-end gateways, mid-tier sensors, and low-power MCUs). Hardware-in-the-Loop Search enables cross-platform optimization by:

• Using a multi-objective search that incorporates cost models or direct measurements from multiple target devices.
• Finding a Pareto-optimal architecture that provides the best accuracy trade-off across all hardware targets.
• Validating that the model's performance is consistent and predictable, avoiding surprises when deploying at scale. This is essential for TinyML MLOps pipelines managing thousands of edge devices.

A foundational application is the creation and refinement of hardware cost models. These are predictive functions (e.g., lookup tables, small neural networks) that estimate latency, memory, and energy for a candidate network. Hardware-in-the-Loop Search provides the ground-truth dataset required to train them:

• Profiling thousands of candidate sub-networks on real hardware to build a mapping from architecture parameters to performance metrics.
• The resulting cost model can then be used in faster, simulation-based NAS (like DARTS or One-Shot NAS) with high confidence, dramatically accelerating the search process after the initial profiling investment.
• This creates a virtuous cycle where cost models improve over time with more profiling data.

Searching for models that are inherently robust to quantization is a major use case. Post-training quantization (PTQ) can cause significant accuracy drops on some architectures. Hardware-in-the-Loop Search can integrate this by:

• Profiling candidates after they have been quantized to 8-bit or even 4-bit integer precision using the target hardware's native integer units.
• Directly measuring the accuracy-latency trade-off of the quantized model, as quantization often speeds up inference.
• This leads to the discovery of quantization-friendly architectures with smoother weight distributions and activation functions that minimize precision loss, a technique known as Quantization-Aware NAS (QA-NAS).

In embedded systems, the ML model is part of a larger pipeline including sensor sampling, signal processing, and power management. Hardware-in-the-Loop Search allows for full-system co-optimization:

• Measuring end-to-end latency from sensor interrupt to prediction output, including pre-processing steps.
• Evaluating power cycles where the model inference wakes the MCU from a deep sleep state, optimizing for total energy per duty cycle.
• Ensuring the model's computational pattern does not conflict with other critical real-time tasks on the same core. This holistic view is essential for building reliable, production-ready TinyML applications for predictive maintenance, keyword spotting, or anomaly detection.

Hardware-Aware Neural Architecture Search is a variant of NAS that incorporates hardware-specific constraints—such as latency, memory footprint, and energy consumption—directly into the search objective. Unlike traditional NAS which optimizes only for accuracy, this approach discovers models optimized for a specific target deployment platform (e.g., a microcontroller or mobile SoC).

• Core Objective: Jointly optimize for task performance (accuracy) and hardware efficiency.
• Key Inputs: A hardware cost model or direct profiling to estimate latency, memory, and power.
• Output: A Pareto-optimal frontier of models representing the best trade-offs between accuracy and hardware metrics.

A hardware cost model is a predictive function used within Hardware-Aware NAS to estimate the resource consumption of a candidate neural network architecture without deploying it. It acts as a fast proxy for actual hardware measurements.

• Types: Can be a lookup table of pre-measured operator costs, a small regression neural network, or an analytical model.
• Purpose: Drastically accelerates the search by avoiding time-consuming on-device profiling for every candidate.
• Limitation: Models can have prediction error, which is why Hardware-in-the-Loop Search is used for ground-truth validation.

Microcontroller NAS (MCU-NAS) refers to hardware-aware neural architecture search methods specifically tailored to the extreme constraints of microcontrollers. It optimizes for metrics irrelevant in cloud contexts.

• Target Metrics: Kilobyte-scale SRAM/Flash usage, microjoule-level energy per inference, and inference latency measured in milliseconds on sub-100 MHz CPUs.
• Unique Challenges: Must account for lack of an OS, fixed-point arithmetic, and severe memory bandwidth limitations.
• Relation to HiL: MCU-NAS often employs HiL search in its final stages to verify that discovered models actually run within the device's physical limits.

Neural Hardware Co-Design is an integrated design philosophy where neural network architectures and hardware accelerator features are jointly optimized. HiL Search is a practical tool enabling this co-design loop.

• Philosophy: The algorithm and the silicon are designed in tandem for peak system-level efficiency.
• Process: 1) Use NAS to find optimal models for a hardware prototype. 2) Use insights to inform next-generation chip design (e.g., which operators to accelerate). 3) Repeat.
• Outcome: Produces highly specialized systems like Google's TPU + Transformer models, where the hardware and software are perfectly matched.

Quantization-Aware NAS (QA-NAS) is a hardware-aware search method that discovers neural network architectures robust to post-training quantization or that are trained with simulated quantization during the search.

• Goal: Find architectures that maintain high accuracy when their weights and activations are converted from 32-bit floats to 8-bit integers (INT8) or lower.
• Method: Integrates quantization simulation (QAT) into the supernet training or candidate evaluation loop.
• Synergy with HiL: A model found via QA-NAS is an ideal candidate for final HiL validation, as its quantized performance is predicted to be stable on the actual integer-arithmetic unit of the target hardware.

Once-For-All (OFA) is a one-shot NAS and training paradigm that produces a single, large supernet from which many efficient, hardware-tailored sub-networks can be extracted without retraining. It is a foundational technique often used with HiL profiling.

• Core Innovation: Weight sharing across a massive space of sub-networks with different depths, widths, and kernel sizes.
• Deployment: After training the supernet, a search algorithm (potentially using a HiL loop) profiles thousands of sub-networks on the target device to select the best one for given constraints.
• Efficiency: Eliminates the need to train every candidate model from scratch, making large-scale HiL search feasible.