Neural Architecture Search (NAS)

The search space is the predefined set of all possible neural network architectures the NAS algorithm can explore. It defines the building blocks (e.g., convolution types, activation functions) and their permissible connections. A well-designed search space balances expressiveness (to find high-performing models) with tractability (to make the search feasible). Common formulations include:

• Cell-based spaces: Search for a repeating computational cell (normal and reduction) that is stacked to form the full network.
• Hierarchical spaces: Define architectures at multiple levels of granularity, from operations within a cell to the macro-structure.
• Chain-structured spaces: Define a sequential chain of layers with optional skip connections.

The search strategy is the optimization algorithm that navigates the search space to discover high-performing architectures. It determines how candidate models are sampled, evaluated, and refined. Key strategies include:

• Reinforcement Learning (RL): A controller network (RNN) generates architecture descriptions, which are trained and rewarded; the controller's policy is updated via policy gradients.
• Evolutionary Algorithms: A population of architectures undergoes selection, mutation, and crossover, mimicking natural evolution.
• Bayesian Optimization: Uses a probabilistic surrogate model to predict architecture performance and guide the search toward promising regions.
• Gradient-Based Methods (e.g., DNAS): Relax the search space to be continuous, allowing architecture parameters to be optimized via gradient descent alongside network weights.

The performance estimation strategy is the method used to assess the quality (e.g., validation accuracy, latency) of a candidate architecture without the prohibitive cost of full training from scratch. This is the primary computational bottleneck in NAS. Efficient strategies include:

• Weight Sharing: A single, over-parameterized supernet is trained; candidate sub-architectures inherit (share) its weights for fast evaluation, as in One-Shot NAS.
• Low-Fidelity Estimation: Evaluating on a subset of data, for fewer epochs, or with a smaller proxy model.
• Learning Curve Extrapolation: Predicting final performance from early training epochs.
• Zero-Cost Proxies: Using heuristics like gradient norm or synaptic saliency that require only a forward/backward pass to score architectures.

A supernet (or one-shot model) is an over-parameterized graph that contains all possible operations and paths defined by the search space. Weight sharing is the technique where all candidate architectures (subgraphs) share the parameters of this supernet. This is the engine behind efficient NAS paradigms:

• The supernet is trained once (or via an iterative process).
• To evaluate a specific architecture, the corresponding path is activated within the supernet, and its performance is approximated using the shared weights.
• This avoids training thousands of models from scratch. Methods like Differentiable NAS (DNAS) and Once-For-All (OFA) rely on this core component.

In Reinforcement Learning (RL)-based NAS, the controller network is a neural network (typically an RNN) that acts as the search strategy's policy. Its role is to sequentially generate architectural hyperparameters (e.g., layer type, filter size).

• The controller samples a candidate architecture string.
• A child network is built from this string, trained, and its validation accuracy becomes the reward.
• This reward is used to compute a policy gradient (e.g., REINFORCE) to update the controller's parameters, increasing the probability of generating high-performing architectures in the future.

Modern NAS extends beyond accuracy to optimize for real-world deployment constraints. This involves augmenting the search objective and estimation strategy:

• Multi-Objective NAS: Optimizes for competing goals simultaneously (e.g., accuracy vs. model size vs. latency), producing a Pareto frontier of optimal trade-offs.
• Hardware-Aware NAS: Directly incorporates deployment target metrics into the search loop. This uses:
  • Latency predictors or on-device measurement.
  • Energy consumption models.
  • Memory footprint estimators. The search algorithm is penalized for architectures that violate these constraints, finding models like EfficientNet that are optimal for specific hardware.

A gradient-based NAS method that formulates the search over discrete architectural choices as a continuous optimization problem. A supernet contains all candidate operations, with architecture parameters made differentiable via a softmax relaxation. This allows the search to be performed using standard gradient descent, making it significantly more efficient than reinforcement learning or evolutionary-based methods. The final discrete architecture is derived by selecting the operation with the highest weight at each choice point.

An efficient NAS paradigm where a single, over-parameterized supernet is trained once. Candidate sub-architectures are evaluated by inheriting weights from this shared supernet, eliminating the need to train each candidate model from scratch. The core challenge is optimization bias: ensuring the shared weight training provides a fair ranking of different sub-networks. Methods like path dropout and sandwich rule training are used to mitigate this bias and improve ranking correlation.

The predefined, structured set of all possible neural network architectures that a NAS algorithm can explore. It defines the building blocks and their allowable connections. Common design dimensions include:

• Cell-based spaces: Search for a computation cell, then stack it.
• Global macro spaces: Search over entire network depth, width, and layer types.
• Operation choices: Convolution type (e.g., 3x3, 5x5, depthwise), activation functions, attention modules. A well-designed search space incorporates prior knowledge to constrain the search to feasible, high-potential regions.

The foundational technique that enables efficient NAS by having all candidate architectures share parameters from a single supernet. Instead of training thousands of independent models, sub-networks are treated as different paths through the supernet. Their performance is estimated using these shared weights, reducing search cost from thousands of GPU days to a few. The accuracy of this performance estimation is critical and is a major focus of one-shot NAS research.

A specialized form of NAS that directly optimizes for deployment constraints on target hardware. The search objective is a composite function, such as Accuracy - λ * Latency. It requires:

• Latency/Efficiency predictors: Either a lookup table, a pre-characterized latency model, or an on-device measurement loop.
• Hardware diversity: Can target mobile CPUs (e.g., ARM), edge TPUs, GPUs, or specialized neural processing units (NPUs).
• Practical outcome: Discovers models that achieve the best accuracy under a strict latency or memory budget, such as models for real-time video processing on smartphones.

A NAS variant that optimizes for multiple, often competing objectives simultaneously. Instead of a single "best" model, the search produces a Pareto frontier—a set of architectures where improving one objective (e.g., accuracy) necessitates worsening another (e.g., model size or energy use). Common objectives include:

• Predictive accuracy
• Inference latency
• Parameter count or memory footprint
• Energy consumption
• Robustness to adversarial examples or distribution shift Optimization methods include scalarization, evolutionary algorithms on Pareto dominance, and multi-gradient descent.