Neural Architecture Search (NAS)

The search space defines the universe of all possible neural network architectures the algorithm can explore. It is parameterized by architectural decisions such as:

• Operator types (e.g., standard convolution, depthwise convolution, pooling).
• Connectivity patterns (e.g., sequential, residual, dense blocks).
• Layer hyperparameters like kernel size and number of channels. For Hardware-Aware NAS, the search space is constrained to exclude architectures that would inherently violate hardware limits, such as those requiring more memory than available on a target microcontroller.

The search strategy is the optimization algorithm that navigates the search space to discover high-performing architectures. Common strategies include:

• Reinforcement Learning (RL-NAS): A controller network learns to generate architectures that maximize a reward based on accuracy and hardware cost.
• Evolutionary Algorithms: A population of architectures is evolved using selection, mutation, and crossover.
• Gradient-Based Methods (e.g., DARTS): The discrete search space is relaxed into a continuous one, allowing architecture parameters to be optimized via gradient descent.
• Random Search: Serves as a simple but often surprisingly effective baseline.

Evaluating each candidate architecture by fully training it is computationally prohibitive. The performance estimation strategy provides efficient proxies for final model quality. Key methods are:

• Weight Sharing / One-Shot NAS: A single supernet encompassing all possible operations is trained once. Candidate sub-architectures are evaluated by inheriting weights from this shared network without retraining.
• Low-Fidelity Estimation: Training on a subset of data, for fewer epochs, or on a smaller proxy model.
• Zero-Cost Proxies: Metrics like gradient norm or synaptic flow that predict trainability from a network's initial state with minimal computation.
• Hardware-in-the-Loop Profiling: Directly measuring latency, energy, or memory on the target device for ground-truth hardware cost.

A critical component for Hardware-Aware NAS, the hardware cost model predicts the on-device efficiency metrics of a candidate architecture. It translates architectural specifications into estimates for:

• Inference Latency (milliseconds)
• Peak Memory Usage (kilobytes of RAM/Flash)
• Energy Consumption (microjoules) Models range from simple lookup tables (LUTs) of pre-measured operator costs to small neural network predictors. This allows the search algorithm to incorporate constraints like latency < 10ms or model size < 512KB directly into its optimization objective.

A supernet is an over-parameterized neural network that instantiates the entire search space. Every possible path and operation defined in the search space exists within this single network. During One-Shot NAS, the supernet is trained with all paths active. To evaluate a specific candidate architecture, the corresponding sub-graph is extracted, and its performance is estimated using the shared weights. This weight sharing mechanism is what makes large-scale NAS feasible, reducing evaluation cost from thousands of GPU-days to a few.

NAS, especially for embedded systems, is inherently a multi-objective optimization problem. The goal is not just maximum accuracy but finding the best trade-off between competing objectives like:

• Model Accuracy
• Inference Latency
• Model Size (Parameters)
• Energy Consumption The search aims to discover architectures on the Pareto frontier, where improving one objective (e.g., lowering latency) necessarily worsens another (e.g., reducing accuracy). The result is a suite of models offering different optimal trade-offs for various deployment scenarios.

This variant directly incorporates hardware-specific constraints—such as latency, memory footprint, and energy consumption—into the search objective. Instead of just maximizing accuracy, the algorithm discovers models optimized for a target deployment platform like a mobile phone or microcontroller.

• Key Constraint Types: Latency (inference time), peak RAM/Flash usage, and energy per inference.
• Search Integration: Constraints are added as penalty terms in the loss function or used to filter candidate architectures.
• Example: MNasNet from Google uses reinforcement learning to optimize for both ImageNet accuracy and real-world latency on Pixel phones.

DARTS is a gradient-based NAS method that reformulates the discrete search problem into a continuous one. It constructs a supernet where the choice between operations (e.g., convolution, pooling) is represented by continuous architecture parameters (alphas).

• Mechanism: The supernet and architecture parameters are jointly optimized via gradient descent.
• Efficiency: Significantly faster than reinforcement learning or evolutionary-based searches.
• Outcome: After training, a discrete architecture is derived by selecting the operation with the highest learned alpha weight at each layer.

This family of methods dramatically reduces search cost by training a single, over-parameterized supernet once. Candidate sub-architectures are evaluated by inheriting weights from this shared supernet without needing individual training.

• Weight Sharing: A single set of supernet parameters approximates the performance of thousands of sub-networks.
• Once-For-All (OFA): A prominent example that trains a supernet to support diverse sub-networks of varying depths, widths, and kernel sizes for different hardware targets.
• Benefit: Enables efficient search across massive design spaces, making NAS feasible on limited computational budgets.

These are the core algorithms that explore the NAS search space.

• Reinforcement Learning NAS (RL-NAS): A controller network (typically an RNN) is trained with policy gradient methods to generate architecture descriptions. The reward is the candidate's validated accuracy.
• Evolutionary NAS: Treats architectures as individuals in a population. It uses mutation (e.g., changing an operation) and crossover (combining parts of two architectures) to evolve better models over generations.
• Comparison: RL-NAS is sample-efficient but complex to train; Evolutionary methods are simple, parallelizable, and often discover high-performing models.

Extends NAS to balance multiple, often competing, goals simultaneously. The aim is to find architectures on the Pareto frontier, where improving one metric (e.g., accuracy) worsens another (e.g., latency).

• Common Objectives: Model accuracy, inference latency, model size (parameters), and energy consumption.
• Search Methods: Uses algorithms like NSGA-II (a genetic algorithm) or incorporates weighted sum losses into gradient-based searches.
• Application: Critical for edge and mobile deployment, where trade-offs between performance and efficiency must be explicitly managed.

Specialized NAS methods targeting the most constrained devices, such as microcontrollers (MCUs). The search optimizes for kilobyte-scale memory, microjoule-level energy, and the absence of advanced operators (e.g., batch normalization).

• Constraints: Extremely tight memory bounds (RAM & Flash) and latency measured in milliseconds.
• Techniques: Often employs hardware-in-the-loop profiling on real MCU dev boards and quantization-aware NAS (QA-NAS) to find models robust to 8-bit or lower precision.
• Goal: Automate the design of networks that are impossible to craft manually for ultra-low-power IoT sensors and wearables.

The search space defines the universe of all possible neural network architectures a NAS algorithm can explore. It is parameterized by variables such as:

• Layer types (e.g., convolution, pooling, attention)
• Connectivity patterns (e.g., skip connections, branching)
• Operator choices (e.g., 3x3 vs. 5x5 convolution, activation function)
• Channel counts (network width) and layer depths (network height) A well-designed search space balances expressiveness (to find good models) with tractability (to make search feasible). Common formulations include cell-based spaces, where a repeating computational cell is optimized, and macro spaces, which define the entire network skeleton.

The search strategy is the core algorithm that navigates the search space to discover high-performing architectures. Major strategy families include:

• Reinforcement Learning (RL): A controller network is trained to generate architectures, receiving a reward based on their trained accuracy.
• Evolutionary Algorithms: A population of architectures is evolved using selection, mutation, and crossover operations.
• Gradient-Based Optimization (e.g., DARTS): The discrete search space is relaxed to be continuous, allowing architecture parameters to be optimized via gradient descent.
• Random Search: Serves as a simple but often surprisingly strong baseline. The choice of strategy directly impacts the computational cost, sample efficiency, and final quality of the discovered architecture.

A performance estimator is a method for predicting the final accuracy or hardware efficiency of a candidate architecture without the prohibitive cost of full training and evaluation. Techniques include:

• Weight Sharing / One-Shot Methods: A single, over-parameterized supernet is trained once. Candidate sub-architectures are evaluated by inheriting weights from this shared network, requiring no retraining.
• Low-Fidelity Estimators: Training on a subset of data, for fewer epochs, or at lower resolution.
• Zero-Cost Proxies: Analytical scores (e.g., gradient norm, synaptic flow) computed from the network's initial state that correlate with trainability.
• Learned Predictors: Small neural networks trained to map architecture encodings to predicted performance. Accurate estimators are critical for making NAS computationally feasible.

A hardware cost model is a predictive function that estimates the on-device performance metrics of a neural network architecture, such as:

• Inference Latency (milliseconds)
• Peak Memory Usage (RAM & Flash in kilobytes)
• Energy Consumption (microjoules) These models enable Hardware-Aware NAS. They can be implemented as:
• Look-up Tables (LUTs): Pre-measured costs of individual operators (e.g., a 3x3 conv layer with 64 channels) on target hardware, summed for a full network.
• Neural Network Predictors: Small MLPs trained to predict cost from an architecture encoding.
• Hardware-in-the-Loop Profiling: Direct, on-device measurement during search (accurate but slower). The model ensures discovered architectures meet strict deployment constraints for microcontrollers or mobile devices.

Multi-Objective NAS optimizes neural architectures across several, often competing, goals simultaneously. The primary objective is typically accuracy, while secondary objectives reflect deployment constraints, such as:

• Model size (parameters)
• Computational complexity (FLOPs)
• On-device latency
• Energy consumption The output is not a single "best" architecture, but a Pareto frontier—a set of architectures where improving one objective (e.g., accuracy) necessarily worsens another (e.g., latency). Search algorithms like NSGA-II (an evolutionary algorithm) are designed to discover and populate this frontier, allowing engineers to select the optimal trade-off for their specific application.

A supernet is an over-parameterized neural network that embeds all possible operations and pathways defined by a NAS search space. Weight sharing is the technique of training this single supernet, then evaluating candidate sub-architectures by inheriting (sharing) the corresponding weights without retraining. This is the foundation of One-Shot NAS methods like DARTS and Once-For-All (OFA). Key Advantages:

• Drastically reduces search cost from thousands of GPU days to a few.
• Enables rapid evaluation of millions of architectures post-search. Key Challenge:
• Weight entanglement, where the performance of a sub-architecture is biased by being co-trained with many others in the supernet, potentially reducing ranking reliability.