Neural Architecture Search (NAS)

The search space defines the universe of possible neural network architectures the NAS algorithm can explore. It is a critical design choice that balances expressiveness and tractability.

• Macro-search spaces define high-level connectivity patterns (e.g., number of layers, types of cells).
• Micro-search (or cell-based) spaces define the internal operations of repeated computational blocks (e.g., choosing between a 3x3 convolution, 5x5 depthwise convolution, or identity operation within a cell).
• Common constraints include maximum model size, latency (e.g., measured on a target device), or FLOPs count. An overly large search space is computationally infeasible to explore exhaustively.

The search strategy is the algorithm that navigates the search space to discover high-performing architectures. Different strategies offer trade-offs between exploration, exploitation, and computational cost.

• Reinforcement Learning (RL): Uses a controller (often an RNN) to generate architecture descriptions, which are trained and rewarded based on their performance.
• Evolutionary Algorithms: Maintain a population of architectures, applying mutation and crossover operations to evolve better designs over generations.
• Gradient-Based Methods: (e.g., DARTS) relax the discrete search space to be continuous, allowing architecture parameters to be optimized via gradient descent alongside network weights.
• Bayesian Optimization: Models the performance landscape to intelligently propose the most promising architectures to evaluate next.

Evaluating every candidate architecture by fully training it is prohibitively expensive. Performance estimation strategies provide cheaper proxies for final accuracy.

• Low-Fidelity Estimation: Training candidates for fewer epochs or on a smaller dataset.
• Weight Sharing / One-Shot Models: Training a single, over-parameterized supernet that contains all possible operations in the search space. Candidate architectures are evaluated as subgraphs of this supernet, sharing its weights, which allows for near-instant ranking.
• Learning Curve Extrapolation: Predicting final performance from early training metrics.
• Hardware-in-the-Loop Profiling: Directly measuring latency, memory, or energy consumption on the target device (e.g., a mobile phone) during the search.

Modern NAS is rarely about accuracy alone. Hardware-aware NAS directly optimizes for metrics critical to on-device deployment, making it a cornerstone of efficient model design.

• Multi-Objective Optimization: The search algorithm optimizes for a joint objective, such as maximizing accuracy while minimizing latency or model size.
• Pareto-Optimal Frontiers: The result is a set of architectures representing the best possible trade-offs (e.g., a model that is 2% more accurate but 20ms slower vs. a slightly less accurate but much faster model).
• Target metrics are integrated directly into the search loop via lookup tables, on-device profilers, or pre-characterized latency models for different operators.

DARTS is a seminal gradient-based NAS method that reformulates architecture search as a continuous optimization problem.

• It introduces continuous architecture parameters (alphas) that represent the strength of connections between operations in a computational cell.
• The search alternates between:
  1. Optimizing network weights via standard gradient descent.
  2. Optimizing the architecture parameters via gradient descent on a validation set.
• After the search, a discrete architecture is derived by selecting the operations with the strongest alpha values.
• DARTS is significantly faster than RL-based methods but can be memory-intensive and prone to performance collapse if not carefully regularized.

One-Shot NAS methods dramatically reduce search cost by weight sharing. A single, trainable supernet encapsulates the entire search space.

• Training the Supernet: The supernet is trained once, typically with path dropout or uniform sampling, to co-optimize the weights for all possible sub-architectures.
• Architecture Search: After training, the search strategy evaluates millions of candidate sub-networks by inheriting weights from the supernet without any additional training, enabling rapid performance estimation.
• Final Training: The discovered optimal architecture is retrained from scratch to obtain its final weights, as the shared weights from the supernet are suboptimal. Popular implementations include ProxylessNAS and FBNet.

A specialized form of NAS where the search algorithm directly optimizes for hardware-specific metrics like latency, power consumption, and memory bandwidth on the target device (e.g., a specific mobile SoC or microcontroller). Instead of just accuracy, the search space includes operations and configurations known to be efficient on the target hardware. Key aspects include:

• Using a latency/power lookup table pre-measured on the target device to guide the search.
• Incorporating hardware-specific primitives like depthwise separable convolutions or quantized operators into the search space.
• The output is an architecture inherently optimized for the deployment environment, reducing the need for post-search compression.

An efficient NAS approach that dramatically reduces computational cost by training a single, over-parameterized supernetwork that encompasses all possible architectures in the search space. During search, candidate architectures are evaluated as subgraphs of this supernetwork by sharing its weights, avoiding training each candidate from scratch. The process involves:

1. Supernet Training: The entire graph is trained once.
2. Architecture Search: Algorithms (e.g., gradient-based, evolutionary) sample and evaluate sub-networks using the shared weights.
3. Final Training: The discovered best architecture is typically retrained from scratch. This method can reduce search time from thousands of GPU days to just a few.

A gradient-based NAS method that formulates the search for discrete architectural components (e.g., which operation to use between two layers) as a continuous optimization problem. It assigns a continuous architecture weight (alpha) to each candidate operation (e.g., convolution, pooling, skip-connect) in a mixed-operation layer. The core idea is:

• The supernetwork output becomes a weighted sum of all operations.
• The search jointly optimizes the standard model weights (w) and the architecture weights (alpha) using bilevel optimization.
• After search, a discrete architecture is derived by retaining the operations with the highest alpha values. DARTS is a foundational one-shot method but can be prone to stability issues and performance collapse.

A family of models discovered and scaled using a principled approach that is closely related to NAS insights. While not strictly a NAS algorithm, EfficientNet demonstrates the power of coordinated scaling. The authors used a NAS search (based on the MnasNet method) to find a good baseline network (EfficientNet-B0). They then applied compound scaling, which uniformly scales the network's depth, width, and input resolution using a fixed set of coefficients. The rule is: depth = α^φ, width = β^φ, resolution = γ^φ, where αβ²γ²≈2 and φ is a user-specified compound coefficient. This method shows that balancing these three dimensions is more effective than scaling just one, a principle often integrated into modern NAS frameworks.

The foundational component of any NAS system, defining the set of all possible neural network architectures the search algorithm can explore. For on-device optimization, this design is critical. Key elements include:

• Macro-architecture: The high-level skeleton (e.g., number of stages, resolution changes).
• Cell-based design: Searching for a optimal computational 'cell' (normal and reduction cells) that is stacked repeatedly to form the full network.
• Operation set: The primitive operations available at each layer (e.g., 3x3 conv, 5x5 depthwise conv, max pool, identity, zero).
• Connectivity patterns: Defining how layers or cells can be connected (e.g., directed acyclic graph within a cell). A well-designed, constrained search space incorporating efficient operations is essential for finding viable edge models.

NAS frameworks that optimize for multiple, often competing, objectives simultaneously. For edge deployment, the primary objective is no longer just validation accuracy. Common objective pairs include:

• Accuracy vs. Latency: Finding architectures on the Pareto frontier that offer the best accuracy for a given latency target.
• Accuracy vs. Model Size (Parameters): Directly targeting memory-constrained devices.
• Accuracy vs. Energy Consumption. Search strategies include:
• Weighted Sum: Combining objectives into a single reward (e.g., Reward = Acc - λ * Latency).
• Pareto-Based Methods: Using evolutionary algorithms to maintain and improve a set of non-dominated solutions.
• Predictor-Based: Training a cheap surrogate model to predict objectives and guide the search efficiently.