Hardware-Aware Neural Architecture Search (NAS)

The search space is defined by architectural parameters that directly impact hardware performance, not just accuracy. This includes:

• Operator types (e.g., depthwise vs. standard convolutions)
• Kernel sizes and filter counts
• Network depth and width
• Skip connection patterns
• Activation functions (e.g., ReLU6 for quantization-friendly ranges) The search algorithm explores combinations of these parameters to find the optimal trade-off.

Instead of using generic FLOPs or parameter counts as proxies, hardware-aware NAS employs predictive cost models that estimate true on-target metrics:

• Latency predictors trained on hardware execution traces.
• Power consumption models based on operator-level energy profiles.
• Memory footprint estimators for peak RAM/VRAM usage. These models allow for rapid evaluation of thousands of candidate architectures without costly on-device deployment for each one. Tools like ProxylessNAS and FBNet pioneered this approach.

Modern hardware-aware NAS uses efficient search strategies to avoid the prohibitive cost of training each candidate from scratch.

• Differentiable Architecture Search (DARTS): Treats the architecture as a set of continuous parameters, optimized via gradient descent alongside network weights.
• One-Shot NAS: Trains a single supernetwork encompassing all possible architectural choices. Candidate sub-networks are evaluated by sharing the supernetwork's weights, enabling orders-of-magnitude faster search. These methods make hardware-in-the-loop optimization feasible within practical timeframes.

The most accurate feedback comes from direct hardware execution. Key techniques include:

• Hardware-in-the-Loop Evaluation: Deploying and profiling top candidate architectures on the actual target device (e.g., a specific smartphone SoC or microcontroller).
• Benchmarking Suites: Using standardized edge benchmarks like MLPerf Tiny to measure latency, accuracy, and energy use under controlled conditions.
• Compiler-Aware Search: Accounting for the optimizations applied by deployment compilers like Apache TVM or TensorFlow Lite, which can significantly alter final performance.

The core output is not a single 'best' model, but a Pareto frontier of optimal architectures representing the best possible trade-offs between competing objectives. For example, a frontier might show:

• Model A: 95% accuracy, 50ms latency.
• Model B: 92% accuracy, 20ms latency.
• Model C: 89% accuracy, 10ms latency. Engineers can then select the specific model from this frontier that best matches their deployment constraints.

Hardware-aware NAS is often combined with model compression techniques in a unified optimization loop. This includes:

• Quantization-Aware NAS: Searching for architectures that are robust to the precision loss from INT8 or FP16 quantization.
• Pruning-Aware NAS: Co-designing the architecture to be inherently sparse, facilitating more effective weight pruning.
• Compiler-Optimized Search: Using the final compiled kernel latency from frameworks like TVM or ONNX Runtime as the optimization target, ensuring the discovered architecture is efficient at the hardware instruction level.

Neural Architecture Search is the overarching automated process for discovering optimal neural network architectures, typically framed as a search problem over a predefined design space. It is the parent methodology of Hardware-Aware NAS.

• Core Mechanism: Uses techniques like reinforcement learning, evolutionary algorithms, or gradient-based methods to iteratively propose, train, and evaluate candidate architectures.
• Objective: Traditionally focused solely on maximizing predictive accuracy on a benchmark dataset (e.g., ImageNet).
• Contrast with Hardware-Aware NAS: Standard NAS does not explicitly model hardware constraints like latency or power, often resulting in architectures too large for edge deployment.

Model Compression is a set of post-training techniques applied to a pre-defined neural network to reduce its computational footprint for deployment. It is a complementary, often subsequent, step to Hardware-Aware NAS.

• Key Techniques: Includes quantization (reducing numerical precision of weights/activations), pruning (removing redundant parameters), and knowledge distillation (training a smaller model to mimic a larger one).
• Relationship to NAS: Hardware-Aware NAS designs efficient architectures from scratch, while compression optimizes existing ones. They are often used in tandem: NAS finds a Pareto-optimal architecture, which is then further compressed via quantization for final deployment.

The Once-For-All (OFA) Network is a pioneering supernet approach that enables hardware-aware NAS without repeated training. A single, large network is trained to contain many smaller sub-networks within its weight-sharing structure.

• Mechanism: The supernet is trained once. The search process then quickly evaluates the performance of different sub-networks (varying in depth, width, kernel size) by sampling from this shared weight pool, without retraining.
• Hardware Constraint Integration: A latency lookup table for the target device is built, allowing the search algorithm to directly filter for sub-networks meeting specific latency or FLOPs targets.
• Impact: Dramatically reduces the computational cost of NAS, making it feasible for edge deployment scenarios.

Differentiable Architecture Search is a gradient-based NAS method that relaxes the discrete architecture search space into a continuous one, allowing optimization via standard gradient descent.

• Core Innovation: Represents the choice of operations between nodes as a continuous mixture (using a softmax over candidate ops). The architecture parameters (alpha) are learned alongside model weights.
• Hardware-Aware Extension: DARTS can be extended to be hardware-aware by adding latency or energy cost terms to the loss function, penalizing architectures that are inefficient on the target hardware.
• Trade-off: While efficient, the continuous relaxation can lead to a discrepancy between the searched continuous architecture and the final discrete one, and the added hardware loss requires accurate cost modeling.

A Hardware Cost Model is a critical, often surrogate, component of Hardware-Aware NAS that predicts the performance metric (e.g., latency, energy) of a candidate neural network architecture on specific target hardware.

• Purpose: Provides a fast, differentiable proxy for expensive on-device measurements during the architecture search loop.
• Types: Can be:
  • Analytical: Based on counting FLOPs, memory access costs, and known hardware parameters.
  • Lookup Table (LUT): Pre-measured latency for each operator (convolution, pooling) on the target hardware, with total latency estimated by summing component costs.
  • Learned (Neural Predictor): A small neural network trained to map an architecture encoding to its predicted latency/energy.
• Accuracy Importance: The fidelity of the cost model directly determines whether the NAS-discovered architecture will meet real-world constraints.

Platform-Aware Optimization is the broader engineering discipline of tailoring software—including ML models—to the specific characteristics of the underlying hardware platform (CPU, GPU, NPU, MCU). Hardware-Aware NAS is a key technique within this field.

• Scope: Encompasses low-level kernel optimization, memory layout tuning, compiler passes (e.g., in Apache TVM), and model-level design choices.
• Hardware Targets:
  • Mobile SoCs (e.g., with NPUs): NAS targets latency and thermal limits.
  • Microcontrollers (MCUs): NAS targets extreme memory (KB of SRAM/Flash) and power (mW) constraints, as seen in TinyML.
• Toolchain Integration: Effective Hardware-Aware NAS requires tight integration with the platform's SDK, compiler (e.g., TensorFlow Lite, PyTorch Mobile), and profiling tools to obtain accurate cost feedback.