Neural Architecture Search (NAS)

The search space defines the universe of possible neural network architectures the NAS algorithm can explore. It is a critical design choice that balances flexibility with tractability.

• Cell-based search spaces define repeating computational blocks (normal and reduction cells) that are stacked to form the full network, as popularized by NASNet and ENAS.
• Macro search spaces allow optimization over the entire network topology, including the number of layers, layer types, and connectivity patterns.
• Hierarchical search spaces combine both levels, enabling efficient exploration of complex architectures. A constrained search space makes the optimization problem feasible but may limit discovery.

The search strategy is the optimization algorithm that navigates the search space to find high-performing architectures. It determines the efficiency and effectiveness of the NAS process.

Key strategies include:

• Reinforcement Learning (RL): Uses an RNN controller to sample architectures, which are trained, and the resulting accuracy is used as a reward to update the controller (e.g., NASNet).
• Evolutionary Algorithms: Maintains a population of architectures, applying mutation and crossover to generate offspring, with selection pressure based on performance (e.g., AmoebaNet).
• Gradient-Based Optimization: Treats the architecture selection as a continuous optimization problem, using techniques like Differentiable Architecture Search (DARTS) to compute gradients with respect to architectural parameters.
• Bayesian Optimization: Models the performance landscape with a surrogate function (e.g., Gaussian Process) to guide the search towards promising regions.

Evaluating a candidate architecture's performance is computationally prohibitive, as it requires full training. Performance estimation strategies are proxies that drastically reduce this cost.

Common methods include:

• Low-fidelity estimation: Training on a subset of data, for fewer epochs, or with a smaller model (proxy network).
• Weight sharing / One-shot models: Training a single, over-parameterized supernet that contains all possible architectures within the search space. Candidate architectures are evaluated as subgraphs of this supernet, sharing its weights, eliminating the need for individual training (e.g., ENAS, DARTS).
• Learning curve extrapolation: Predicting final performance from the early stages of training.
• Surrogate models: Training a separate model (e.g., a regressor) to predict architecture performance based on its encoding.

NAS algorithms must balance exploration (searching new regions of the architecture space) with exploitation (refining known high-performing regions). This is a core challenge in search strategy design.

• Exploration-heavy strategies (e.g., random search early in evolution) help avoid local optima and discover novel building blocks.
• Exploitation-heavy strategies (e.g., fine-tuning via gradient descent in DARTS) efficiently optimize within a promising subspace. Poor balance can lead to premature convergence on suboptimal architectures or excessive, wasteful computation. Techniques like epsilon-greedy policies in RL or temperature parameters in evolutionary algorithms explicitly manage this trade-off.

The primary historical barrier to NAS has been extreme computational cost. Early RL-based searches required thousands of GPU days. Modern research focuses intensely on efficiency.

Efficiency gains come from:

• Weight sharing in one-shot models, reducing search cost to a few GPU days.
• Performance predictors that avoid expensive training.
• Progressive search techniques that start with simple cells and increase complexity.
• Hardware-in-the-loop search, where latency or energy consumption on target hardware (e.g., a mobile phone) is directly optimized as part of the objective, leading to practical hardware-aware NAS.

A key objective of NAS is to discover architectures that generalize well across datasets and tasks, not just perform well on the single proxy task used during search.

• Task-Agnostic Search: Searching on a large, general dataset (like ImageNet) often yields architectures that transfer effectively to many other vision tasks.
• Zero-shot Proxies: Research into metrics that predict generalization without any training (e.g., based on gradient signals or network connectivity) to accelerate cross-task transfer.
• Meta-Learning for NAS: Using meta-learning to learn a search strategy that can quickly adapt to new datasets with minimal computational overhead, moving towards few-shot NAS. Failure to generalize results in a discovered architecture that is overfitted to the search conditions.

One-Shot NAS is an efficiency paradigm where the search is conducted by training a single, over-parameterized supernetwork (or one-shot model) that encompasses all possible architectures in the search space. Weight sharing is its core technique:

• All candidate sub-architectures share the weights of the supernetwork.
• The search algorithm evaluates candidate architectures by activating different paths within this shared weight supernet.
• This avoids the prohibitive cost of training each candidate architecture from scratch. Methods like ENAS (Efficient Neural Architecture Search) popularized this approach, making NAS feasible on limited hardware.

Multi-Objective NAS extends the goal beyond just predictive accuracy. It simultaneously optimizes for multiple, often competing, objectives such as:

• Model Size (Number of parameters)
• Computational Latency (Inference time)
• Energy Consumption
• Memory Footprint This is critical for deployment on edge devices and in production environments where resources are constrained. Search algorithms like NSGA-II (Non-dominated Sorting Genetic Algorithm II) are used to find a Pareto front of architectures representing optimal trade-offs between the objectives.