Hardware-Aware Neural Architecture Search (HW-NAS)

The search space defines the universe of possible neural network architectures the NAS algorithm can explore. In HW-NAS, this is constrained by hardware compatibility.

• Key Elements: Includes choices for operation types (e.g., conv3x3, depthwise-separable conv, MBConv blocks), network depth, layer width (channel count), kernel sizes, and skip connection patterns.
• Hardware-Aware Design: The space is often pre-filtered to exclude operations poorly supported by the target hardware (e.g., avoiding large group convolutions on some NPUs). It may include hardware-native blocks like Tensor Cores or NPU-accelerated activations.
• Example: For a mobile CPU, the search space might prioritize MobileNetV2-style inverted residual blocks over standard convolutions to leverage efficient depthwise operations.

The search strategy is the algorithm that navigates the search space to find high-performing architectures. It must efficiently balance exploration with the high cost of evaluating candidates on hardware.

• Common Strategies:
  • Reinforcement Learning (RL): An RNN controller samples architectures, trained with a reward based on accuracy and hardware cost.
  • Evolutionary Algorithms: Populations of architectures undergo mutation and crossover, selected by a fitness function combining accuracy and latency.
  • Differentiable NAS (DNAS): Formulates the search as a continuous optimization problem using architecture weights and the Gumbel-Softmax trick, allowing efficient gradient-based search.
• Hardware Integration: The strategy's sampling or mutation steps are biased by a predictive model of hardware performance to avoid exploring inefficient regions of the space.

This component provides fast, low-cost estimates of hardware performance metrics (latency, power, memory) for a candidate architecture without full on-device execution.

• Methods:
  • Look-Up Tables (LUTs): Pre-measure latency/energy for each atomic operation (e.g., 3x3 conv on 56x56x32 input) on target hardware. The total cost is summed from the LUT.
  • Analytical Cost Models: Use theoretical formulas based on FLOPs, memory bandwidth, and cache behavior to estimate runtime.
  • Neural Predictors: A small regression model (e.g., MLP) is trained on (architecture, measured_latency) pairs to predict costs for novel architectures.
• Critical Role: The estimator is embedded in the search loop's objective function, enabling the search to directly optimize for hardware efficiency.

The objective function is the mathematical goal the search algorithm optimizes. In HW-NAS, it is a multi-objective function combining task performance and hardware cost.

• Standard Form: Often expressed as L = L_task(A, W) + λ * C_hardware(M), where:
  • L_task is the primary loss (e.g., cross-entropy).
  • C_hardware is the cost from the performance estimator (e.g., latency in ms).
  • λ is a Lagrange multiplier balancing the trade-off.
• Multi-Objective Optimization: Advanced methods use Pareto optimization to find a frontier of architectures representing the best accuracy-efficiency trade-offs, rather than a single solution for one λ value.
• Example Objective: Minimize: Cross-Entropy Loss + 0.1 * (Predicted Latency on Snapdragon 8 Gen 3)

This is the process for training and validating candidate architectures to obtain their final accuracy, which is fed back to the search strategy.

• Weight Sharing / One-Shot NAS: A supernet containing all possible operations is trained once. Candidate architectures are evaluated as sub-networks of this supernet using shared weights, drastically reducing evaluation cost from days to seconds.
• Progressive Training: Promising architectures from the one-shot evaluation may undergo standalone fine-tuning for a few epochs on a validation set to get a more accurate performance ranking.
• Hardware-in-the-Loop Validation: The final top-ranked architectures are compiled and benchmarked on real target devices (e.g., a mobile phone, Raspberry Pi) to verify the performance estimator's predictions and obtain final deployment metrics.

This component provides the critical bridge between the abstract NAS framework and the physical deployment hardware. It is responsible for accurate profiling and final deployment.

• Profiling Tools: Integrates with hardware-specific SDKs and profilers (e.g., Qualcomm SNPE, MediaTek NeuroPilot, ARM Compute Library, NVIDIA TensorRT) to generate the data for the Performance Estimator's LUT or predictor model.
• Compiler Awareness: Understands the target hardware's compilation stack. The search may be aware of compiler optimizations like operator fusion that change the effective cost of certain operation sequences.
• Deployment Output: The final output of a HW-NAS system is not just an architecture definition, but often a fully compiled model file (e.g., .tflite, .onnx) optimized for the target accelerator, ready for on-device inference.

Neural Architecture Search (NAS) is the automated process of discovering high-performing neural network architectures for a given task, replacing manual design. It treats the network's structure—layer types, connections, and hyperparameters—as a search space optimized by a controller algorithm (e.g., reinforcement learning, evolutionary algorithms, or gradient-based methods). The goal is to find architectures that maximize accuracy, often under constraints like model size or computational cost, forming the foundation for more specialized searches like HW-NAS.

Differentiable Neural Architecture Search (DNAS) is a gradient-based NAS method that formulates the search for an optimal sub-network as a continuous optimization problem. Instead of discrete sampling, it uses a supernet containing all candidate operations, with architectural choices represented by continuous architecture parameters. These parameters are optimized via standard gradient descent alongside model weights, making the search significantly faster than reinforcement learning-based methods. DNAS is frequently used in HW-NAS to efficiently explore hardware-efficient building blocks like mixed-precision operators or kernel sizes.

Automated Model Compression (AMC) is a framework that uses reinforcement learning or other search algorithms to automatically determine the optimal compression policy for each layer of a pre-trained network. Unlike HW-NAS, which searches for a new architecture, AMC takes an existing model and decides layer-by-layer parameters such as:

• Pruning ratio
• Quantization bit-width
• Whether to apply low-rank factorization

The agent is rewarded for maximizing compression (reducing FLOPs, memory) while minimizing accuracy loss, creating a hardware-aware compression schedule without manual per-layer tuning.

The Compression-Accuracy Pareto Frontier represents the set of optimal model configurations where no further compression can be achieved without sacrificing accuracy, and vice-versa. In HW-NAS, the search objective is to discover architectures that lie on this frontier for a specific hardware metric (e.g., latency, energy). Analyzing the frontier helps engineers make informed trade-offs:

• Knee points indicate configurations offering the best accuracy gain per unit of compression.
• The frontier is defined by evaluating hundreds of candidate architectures during the NAS process.
• It is a crucial tool for feedback-driven scheduling, guiding how aggressive compression should be for a given performance target.

Hardware-Aware Compression encompasses all model optimization techniques co-designed with or specifically targeting the characteristics of underlying silicon. HW-NAS is a subset of this field. Key aspects include:

• Targeting specific hardware primitives: Optimizing for efficient operations on NPUs (e.g., 4-bit integer dot products) or mobile SoCs (e.g., depthwise convolutions).
• Incorporating hardware feedback: Using direct measurements (latency, power) or proxy models (latency/energy lookup tables) to guide the compression search.
• End-to-end optimization: Considering the full stack from algorithm down to memory bandwidth and cache hierarchy, not just theoretical FLOPs.

In HW-NAS, Multi-Objective Optimization is the mathematical framework for balancing competing goals during the architecture search. The search algorithm must optimize for multiple, often conflicting, objectives simultaneously. A common formulation is: Objective = Accuracy - λ * (Latency + β * Energy) Where λ and β are scaling factors. Techniques include:

• Weighted Sum Method: Combining objectives into a single scalar (as above).
• Pareto Optimization: Maintaining a population of architectures that represent different trade-offs (the Pareto frontier).
• Constraint Optimization: Treating one objective (e.g., latency < 10ms) as a hard constraint while maximizing another (accuracy).