Hardware-Aware Neural Architecture Search

The core innovation of HA-NAS is its multi-objective search objective. Instead of solely maximizing validation accuracy, the search algorithm optimizes a composite function that includes one or more hardware performance metrics. Common objectives include:

• Minimizing inference latency (measured in milliseconds).
• Reducing peak memory consumption (measured in megabytes).
• Lowering energy consumption (measured in joules).
• Maximizing throughput (samples processed per second). The search balances these against task accuracy, often resulting in a Pareto-optimal frontier of architectures representing the best trade-offs.

Directly measuring hardware metrics during search is computationally prohibitive. HA-NAS systems rely on performance proxies—fast, approximate models of hardware behavior. Key proxy types include:

• Analytical predictors: Lightweight formulas estimating latency or FLOPs based on operator characteristics.
• Neural predictors: Small ML models trained to predict a metric (e.g., latency) from an architecture encoding.
• Zero-cost proxies: Heuristics like synaptic saliency or gradient norm that correlate with final trained performance without any training.
• Look-up tables (LUTs): Pre-measured latencies for individual operators (e.g., a 3x3 convolution) on target hardware, summed for a full network.

The search space—the set of all possible architectures—is constrained by the target hardware's capabilities. This hardware-aware design prevents the discovery of architectures that are theoretically efficient but impractical to deploy. Constraints include:

• Operator support: Including only layers with optimized kernels on the target accelerator (e.g., specific depthwise convolutions for mobile NPUs).
• Memory alignment: Ensuring tensor dimensions align with hardware requirements for efficient memory access.
• Kernel fusion opportunities: Designing primitive operations that can be fused into a single, efficient kernel to reduce overhead.
• Data type support: Limiting operations to the numerical precisions (e.g., INT8, FP16) natively supported by the hardware.

Modern HA-NAS often uses Differentiable Neural Architecture Search (DNAS). A continuous supernet contains all candidate operations, connected via learnable architecture parameters (alphas). The training loss is augmented with a hardware regularization term: Total Loss = Task Loss (e.g., Cross-Entropy) + λ * Hardware Loss (e.g., Predicted Latency). The coefficient λ controls the trade-off. During search, gradient descent optimizes both the network weights and the alphas. The final discrete architecture is derived by selecting the operations with the highest alpha values, inherently favoring hardware-efficient choices.

The Once-For-All (OFA) paradigm trains a single supernet that supports many sub-networks of varying depths, widths, and kernel sizes. For HA-NAS, this supernet is profiled across a platform-aware search space. At deployment, an efficient search (e.g., using a neural predictor) finds the optimal sub-network for a specific device's latency or memory budget. This allows a single trained model to serve diverse hardware targets—from high-end servers to edge microcontrollers—without retraining, decoupling model training from hardware-aware specialization.

HA-NAS is frequently combined with other model compression techniques in a unified search, known as AutoML for Compression. The search space is expanded to include:

• Quantization choices: Searching for the optimal bit-width (e.g., 8-bit, 4-bit) per layer or group of layers.
• Sparsity patterns: Co-searching architectural connectivity and weight pruning strategies.
• Compound scaling factors: Jointly optimizing network depth, width, and input resolution (as in EfficientNet) under hardware constraints. This holistic approach finds architectures that are inherently efficient and further optimized through precision reduction and sparsity, pushing the Pareto frontier further than standalone techniques.

HW-NAS is critical for deploying models on resource-constrained devices like smartphones, IoT sensors, and drones. The search directly optimizes for:

• Inference latency (e.g., < 20ms for real-time video)
• Peak memory usage to fit within tight RAM budgets
• Energy consumption to extend battery life

Frameworks like Google's MobileNetV3 and Apple's EfficientNets were discovered using HW-NAS, targeting specific smartphone processors. The search space includes mobile-friendly operations like depthwise separable convolutions and squeeze-and-excitation blocks.

This application tailors models for dedicated inference chips (NPUs, TPUs, FPGAs). Each accelerator has unique performance characteristics:

• Memory hierarchy and cache sizes influence layer choices.
• Supported operation set (e.g., specific convolution sizes, activation functions).
• Data movement costs between compute units.

HW-NAS uses hardware-in-the-loop profiling or accurate latency/power lookup tables to evaluate candidates. For example, searches for Google's Edge TPU avoid operations it cannot execute efficiently, maximizing throughput per watt.

Enterprise deployments require balancing competing constraints. HW-NAS performs multi-objective optimization, finding architectures on the Pareto frontier that trade off:

• Model accuracy (e.g., Top-1 %)
• Inference speed (Frames Per Second or queries/sec)
• Model size (MBs for over-the-air updates)
• Energy use (Joules per inference)

Tools like Facebook's ChamNet and Once-For-All (OFA) network exemplify this, providing a spectrum of models from a single search to suit different service-level agreements (SLAs) and hardware tiers within a data center.

HW-NAS enables adaptive models that can be fine-tuned or reconfigured directly on edge devices. Use cases include:

• Personalization: Lightweight architecture search to adapt a base vision model to a user's specific environment.
• Domain Shift Mitigation: Searching for a small adapter network when sensor data distribution changes.
• Federated Learning: Efficiently searching for client-specific architectures in a federated setup without centralizing data.

This relies on ultra-efficient search algorithms like zero-cost proxies (e.g., grad_norm, synaptic flow) that predict architecture quality with minimal computation, feasible on edge hardware.

Companies developing AI software for a heterogeneous hardware ecosystem use HW-NAS to generate a family of models from a single search. The goal is one training pipeline that yields optimized variants for:

• High-end servers (max accuracy)
• Mid-range laptops (balanced performance)
• Entry-level mobile phones (minimum viable model)

The Once-For-All (OFA) paradigm is seminal here. A single supernet is trained, supporting sub-networks of varying depths, widths, and kernel sizes. HW-NAS then quickly extracts the best sub-architecture for each target platform's latency/accuracy profile.

HW-NAS directly contributes to energy-efficient AI by making power consumption a first-class optimization target. Applications include:

• Data Center Efficiency: Searching for models that maintain accuracy while reducing petaFLOPs per training run.
• Carbon-Aware Computing: Designing models whose inference energy footprint aligns with available renewable energy supply.
• Lifecycle Cost Reduction: For large-scale deployments (e.g., millions of devices), a few milliwatts saved per inference translates to massive operational savings and reduced environmental impact.

Metrics like Energy-Delay Product (EDP) are incorporated into the search reward function to find truly Pareto-optimal architectures for sustainability.

Neural architecture search (NAS) is the automated process of discovering high-performing neural network architectures for a given dataset and task. It treats the network design—layer types, connections, hyperparameters—as a search problem.

• Core Objective: Automate architecture engineering, traditionally a manual and expert-driven process.
• Common Search Strategies: Include reinforcement learning, evolutionary algorithms, and gradient-based methods (e.g., Differentiable NAS).
• Key Challenge: The immense computational cost of training and evaluating thousands of candidate architectures, which hardware-aware NAS aims to mitigate by incorporating efficiency metrics early in the search.

Multi-objective NAS extends standard NAS by optimizing for several, often competing, objectives simultaneously. Instead of just accuracy, it seeks architectures that balance multiple constraints.

• Typical Objectives: Model accuracy, inference latency, memory footprint, energy consumption, and parameter count.
• Output: A Pareto-optimal frontier of architectures, where improving one metric worsens another. Hardware-aware NAS is a specialized form of multi-objective search where the secondary objectives are direct hardware performance metrics.
• Search Techniques: Uses algorithms like NSGA-II (a genetic algorithm) or methods that scalarize multiple losses into a single weighted objective.

The Once-For-All (OFA) network is a training paradigm and supernet designed to support numerous sub-networks of varying dimensions (depth, width, kernel size) from a single set of shared weights.

• Core Idea: Train a large supernet once, then extract many smaller, efficient sub-networks without retraining.
• Hardware-Aware Link: OFA enables rapid hardware-aware specialization. After profiling target devices (e.g., mobile CPU, edge GPU), the best-performing sub-network for that device's latency/energy budget can be selected instantly.
• Efficiency: Eliminates the need to run a full NAS search from scratch for each new hardware target, making deployment across heterogeneous devices practical.

A zero-cost proxy is a heuristic metric that estimates the quality or efficiency of a neural architecture without requiring any training. It uses information from a few forward/backward passes or initializations.

• Purpose: Enable ultra-fast pruning of the NAS search space by quickly ranking candidate architectures.
• Examples: Metrics like gradient norm, synaptic saliency (SNIP), Fisher information, or jacobian covariance.
• Hardware-Aware Application: Proxies can be extended to estimate hardware metrics. For instance, a proxy might predict a layer's latency on a specific accelerator based on its operator mix and data flow, guiding the search toward hardware-efficient designs with minimal profiling overhead.

Differentiable NAS (DNAS) formulates the architecture search space as a continuous, differentiable supernet. Architecture parameters (e.g., which operation to use at a layer) are relaxed into continuous variables and optimized via gradient descent alongside the model weights.

• Efficiency Advantage: Significantly faster than reinforcement learning or evolutionary-based NAS, as it avoids training thousands of discrete models from scratch.
• Hardware Integration: Hardware-aware DNAS incorporates latency or energy look-up tables (LUTs) or neural predictors into the differentiable loss function. The search directly optimizes the weighted sum of task loss (e.g., cross-entropy) and a differentiable proxy of the hardware cost.

Automated machine learning (AutoML) aims to automate the end-to-end process of applying machine learning to real-world problems. Hardware-aware NAS is a specialized component within the broader AutoML stack.

• Broader Scope: AutoML includes automated data preprocessing, feature engineering, model selection, hyperparameter optimization (HPO), and model evaluation.
• Integration Point: In a full AutoML pipeline, hardware-aware NAS would be the stage responsible for generating the model architecture, constrained by the deployment environment's specifications provided by the user or system.
• Goal: Democratize ML and reduce development time, while hardware-aware NAS ensures the final model is not just accurate but also deployable.