Neural Architecture Search (NAS)

The search space defines the set of all possible neural network architectures the NAS algorithm can explore. It is a critical design choice that balances flexibility with tractability.

• Cell-based search spaces are common, where the algorithm discovers a repeating computational cell (e.g., a Normal or Reduction Cell) that is stacked to form the final network (e.g., NASNet, ENAS).
• Macro search spaces allow the algorithm to define the entire network's high-level structure, including the number of layers, layer types, and connectivity.
• Hierarchical search spaces combine both, searching over motifs at multiple scales. Constraining the search space with hardware-aware primitives (e.g., depthwise separable convolutions, efficient activation functions) is essential for discovering edge-optimized models.

The search strategy is the algorithm that navigates the search space to sample and evaluate candidate architectures. The choice drastically impacts computational cost and result quality.

• Reinforcement Learning (RL): Early NAS methods used an RL controller (e.g., RNN) to generate architecture descriptions, trained with a reward signal based on the trained model's accuracy (e.g., NAS by Google Brain). This is computationally expensive.
• Evolutionary Algorithms: Population-based methods that use mutation and crossover operations to evolve architectures over generations, selecting based on fitness (e.g., accuracy, latency).
• Gradient-Based Optimization: Differentiable NAS (DNAS) relaxes the discrete search space to be continuous, allowing the use of gradient descent to jointly optimize architecture parameters and model weights (e.g., DARTS). This significantly improves search efficiency.

Performance estimation is the process of evaluating a candidate architecture's quality (e.g., accuracy, latency) without fully training it to convergence, which is the major computational bottleneck in NAS.

• Lower-fidelity estimates train candidates for fewer epochs or on a subset of data to rank them quickly.
• Weight sharing is a key innovation where a single supernet (or one-shot model) encapsulates all possible architectures in the search space. Candidate architectures are evaluated as sub-networks of this supernet, sharing its weights, eliminating the need for individual training from scratch (e.g., ENAS, ProxylessNAS).
• Predictor-based methods train a surrogate model (e.g., a neural network or Gaussian process) to predict the final performance of an architecture based on its encoding, amortizing the cost after initial data collection.

For edge deployment, NAS is explicitly constrained by device limitations. Hardware-aware NAS directly optimizes for metrics beyond accuracy.

• Multi-objective search targets a Pareto-optimal frontier balancing accuracy with inference latency, energy consumption, or model size.
• The search incorporates hardware feedback loops: Candidate architectures are profiled on the target hardware (or an accurate simulator/emulator) to obtain real latency/energy measurements, which guide the search.
• This leads to the discovery of architectures with characteristics like:
  • Optimal kernel sizes and channel widths for a specific CPU/GPU/NPU.
  • Layer types and connectivity that minimize memory bandwidth.
  • Operations that leverage specialized hardware instructions (e.g., INT8 quantization support).

One-Shot NAS methods represent the state-of-the-art in efficiency. They decouple the search process into two phases:

1. Supernet Training: A single, over-parameterized network containing all candidate operations is trained once.
2. Architecture Search: Using the trained supernet weights, an optimal sub-network is identified via a much cheaper search (e.g., using evolutionary algorithms or gradient methods).

Differentiable NAS (DNAS), a subset of one-shot methods, formulates the search as a continuous optimization problem. It assigns architecture distribution parameters (e.g., using a Gumbel-Softmax relaxation) that are jointly optimized with network weights via standard backpropagation. After training, the discrete architecture is derived by selecting the operation with the highest learned probability at each choice point.

When applied to edge constraints, NAS produces model families that define modern efficient AI. These models are not just compressed versions of large networks but are natively designed for efficiency.

• MobileNetV3: Discovered through platform-aware NAS combining search with the NetAdapt algorithm for layer-wise resource adjustment.
• EfficientNet: Uses compound scaling where NAS finds a baseline model (EfficientNet-B0), which is then uniformly scaled up in depth, width, and resolution using a fixed set of coefficients.
• ProxylessNAS: Conducts hardware-aware search directly on the target task and hardware, eliminating the need for a proxy dataset, and can search for specialized architectures for mobile CPUs or mobile GPUs.
• The core result is a Pareto-optimal model that provides the best possible accuracy for a given latency or model size budget on the target edge device.

The search space defines the universe of possible neural network architectures an NAS algorithm can explore. For edge AI, this space is constrained by hardware-aware metrics like latency, memory footprint, and energy consumption. Common design choices include:

• Cell-based search: Discovering repeating computational blocks (normal and reduction cells).
• Macro-architecture search: Optimizing the high-level skeleton of network depth, width, and connection patterns.
• Operator search: Selecting from a predefined set of layer types (e.g., standard convolution, depthwise convolution, inverted residual block). The design of the search space is critical, as an overly large or unconstrained space makes the optimization problem intractable.

The search strategy is the optimization algorithm that navigates the search space to find high-performing architectures. Key strategies include:

• Reinforcement Learning (RL): Uses an RNN controller to sample architectures, which are then trained and rewarded based on validation accuracy. The controller's policy is updated to generate better architectures.
• Evolutionary Algorithms: Maintains a population of architectures, applying mutation (e.g., changing a layer type) and crossover to generate offspring, selecting the fittest based on a fitness function (e.g., accuracy, latency).
• Gradient-Based Optimization: Treats the architecture selection as a continuous optimization problem (e.g., using DARTS - Differentiable Architecture Search). Architecture parameters are learned via gradient descent alongside network weights, making the search significantly faster.

Evaluating each candidate architecture by training it to convergence is computationally prohibitive. Performance estimation strategies provide low-cost proxies for final accuracy:

• Weight Sharing / One-Shot Models: A single, over-parameterized supernet encapsulates all possible architectures in the search space. Sub-networks (child models) inherit weights from the supernet, allowing their performance to be estimated after minimal training or even without any additional training (zero-cost proxies).
• Low-Fidelity Estimation: Training candidates for a reduced number of epochs, on a smaller dataset, or at lower resolution.
• Learning Curve Extrapolation: Predicting final performance from the early stages of training. The choice of estimator creates a critical trade-off between search speed and the reliability of the architecture rankings.

Standard NAS optimizes primarily for accuracy. Multi-Objective NAS simultaneously optimizes for multiple, often competing, constraints essential for edge deployment. The search aims to find architectures on the Pareto frontier, where no objective can be improved without degrading another. Common objectives include:

• Model Accuracy (primary task performance).
• Inference Latency (measured on target hardware).
• Parameter Count / Model Size.
• Energy Consumption (in millijoules per inference).
• Memory Bandwidth. Optimization methods include weighted sum approaches, evolutionary algorithms with Pareto ranking, and constrained optimization where latency/size is a hard budget.

A specialized form of Multi-Objective NAS where the search directly incorporates measurements or predictive models from the target deployment hardware. Instead of relying on proxy metrics like FLOPs, it uses:

• On-device profiling: Measuring actual latency and power for sampled architectures on the target chip (e.g., a mobile CPU, NPU, or microcontroller).
• Hardware performance predictors: Learned models that predict latency/energy for a given architecture description, avoiding repeated on-device compilation and profiling. This ensures the discovered network is not just theoretically efficient but is optimized for the specific cache hierarchy, parallel units, and memory subsystem of the edge device.

A pioneering weight-sharing NAS technique that decouples training from search. A single OFA supernet is trained once to support an enormous number of sub-networks varying in:

• Depth (number of layers).
• Width (number of channels).
• Kernel size.
• Resolution (input image size). After training, a specialized search algorithm (e.g., using a latency lookup table) can instantly extract thousands of hardware-tailored sub-networks without any retraining. This provides a practical, production-ready pipeline for serving different optimal architectures for diverse edge devices from one foundational model.