One-Shot NAS

The core efficiency mechanism. Instead of training thousands of independent models, a single over-parameterized supernet is constructed to contain all candidate architectures as subnetworks. These subnetworks share the supernet's weights, allowing for the evaluation of an architecture's potential by simply activating its specific path through the network. This reduces search cost from thousands of GPU days to a few, as performance is estimated via weight inheritance rather than training from scratch.

Transforms the discrete architecture selection problem into a continuous optimization task. The search space is relaxed using architecture parameters (often denoted as alpha, α) that represent the choice between operations (e.g., 3x3 convolution, 5x5 convolution, identity). A continuous relaxation, such as a softmax over these parameters, allows the use of standard gradient descent to jointly optimize both the supernet's shared weights and the architecture parameters. This is the foundational principle behind methods like DARTS (Differentiable ARchiTecture Search).

One-Shot NAS strictly separates its workflow into two distinct phases:

• Search Phase: The supernet is trained with the relaxed architecture parameters. The goal is to discover the optimal architecture topology, not to achieve peak accuracy. The output is a discrete architecture derived from the optimized α parameters (e.g., by retaining the operation with the highest α).
• Retrain Phase: The discovered architecture is constructed from scratch and trained independently on the target dataset. This final training achieves the model's full accuracy potential, free from the constraints and approximations of the weight-sharing supernet.

A critical challenge is supernet ranking correlation: the performance ranking of subnetworks within the supernet must correlate highly with their performance if trained in isolation. The search algorithm's success depends on this property. Techniques to improve ranking consistency include:

• Uniform path sampling during supernet training to ensure fair optimization.
• Architecture regularization to prevent certain paths from dominating.
• Single-path one-shot methods that train only one random subnetwork per batch, reducing interference.

Easily extends to multi-objective search. The search objective can be modified to include real-world deployment constraints alongside accuracy. By incorporating latency, energy consumption, or model size into the loss function, the gradient-based search naturally discovers architectures optimized for specific hardware (e.g., mobile phones, edge TPUs). This is achieved by using a look-up table or a neural predictor to estimate the cost of each operation or subnetwork during the search.

Despite its efficiency, One-Shot NAS introduces specific trade-offs:

• Supernet Overfitting: The supernet can overfit to the validation set used in the search phase, harming generalization.
• Discretization Gap: The process of deriving a discrete architecture from continuous α parameters can lead to a performance drop versus the relaxed version.
• Memory Overhead: The supernet can be significantly larger than any single subnetwork, requiring substantial GPU memory.
• Search Space Design Bias: Performance is ultimately bounded by the pre-defined, human-designed search space of operations and connections.

Weight sharing is the foundational technique that makes One-Shot NAS computationally feasible. Instead of training each candidate architecture from scratch, all sub-networks (child models) inherit and share parameters from a single, over-parameterized supernet. This allows for the evaluation of thousands of architectures by simply activating different paths within the supernet, amortizing the training cost.

• Core Mechanism: A supernet is trained once. Architectures are evaluated by sampling sub-graphs without independent training.
• Key Benefit: Reduces search cost from thousands of GPU days to a few.
• Primary Challenge: Requires careful design of the supernet training schedule and path sampling strategy to avoid bias and ensure fair ranking of sub-networks.

Differentiable Architecture Search (DARTS) is a seminal gradient-based NAS method and a direct precursor to many One-Shot approaches. DARTS continuously relaxes the discrete search space, making architectural choices (e.g., which operation to use between two nodes) soft and differentiable. This allows the architecture parameters to be optimized via standard gradient descent alongside the model weights.

• Connection to One-Shot NAS: DARTS trains a single supernet with continuous relaxation. The final architecture is derived by discretizing the learned continuous weights (e.g., choosing the operation with the highest weight).
• Limitation: The continuous relaxation can lead to a discretization gap, where the performance of the derived discrete architecture does not match its ranking during the supernet optimization.

Hardware-Aware NAS is an optimization paradigm that incorporates target deployment constraints directly into the architecture search objective. Instead of solely maximizing accuracy, the search optimizes for a multi-objective trade-off, such as accuracy versus latency, memory footprint, or energy consumption on specific hardware (e.g., mobile CPUs, edge TPUs).

• Integration with One-Shot NAS: Hardware metrics can be profiled for each sub-network within the supernet. The search algorithm then uses these efficiency proxies to guide the selection toward architectures that satisfy the hardware constraints.
• Practical Outcome: Enables the automated design of models like MobileNetV3 or EfficientNet, which are optimized for real-world deployment, not just benchmark accuracy.

The search space is the formally defined universe of all possible neural network architectures that a NAS algorithm can explore. For One-Shot NAS, the search space must be encodable within a supernet. It defines the building blocks (e.g., convolutional operation types, kernel sizes, skip connections) and how they can be combined.

• Common Designs:
  • Cell-based search spaces: Search for a optimal computation cell (normal and reduction), then stack it repeatedly.
  • Hierarchical search spaces: Allow macro-architecture choices (number of layers, channels) and micro-architecture choices (per-layer operations).
• Critical Impact: The expressiveness and size of the search space directly limit the potential of the discovered architecture. A poorly designed space cannot yield a state-of-the-art model.

Zero-cost proxies are metrics used to estimate the quality or trainability of a neural architecture without any training, often based on properties of the network at initialization. They are used to prune the search space in One-Shot NAS by quickly identifying and discarding poorly conditioned candidates before expensive supernet training or evaluation.

• Examples:
  • grad_norm: The norm of gradients after a single mini-batch.
  • synflow: A measure of synaptic flow based on a single forward/backward pass.
  • NASWOT: Uses the number of linear regions in the input space.
• Utility: While not perfectly correlated with final trained accuracy, these proxies allow for orders-of-magnitude faster preliminary ranking, making large search spaces tractable.

Evolutionary NAS is an alternative search methodology to One-Shot and gradient-based methods. It uses evolutionary algorithms to maintain a population of architectures, which are iteratively improved through selection, mutation (random architectural changes), and crossover (combining parts of two architectures).

• Contrast with One-Shot NAS: Evolutionary methods typically train each candidate from scratch (or from a checkpoint), making them vastly more computationally expensive but less prone to the weight-sharing bias that can affect One-Shot rankings.
• Modern Hybrids: State-of-the-art approaches often use One-Shot NAS as a performance predictor within an evolutionary loop, combining the efficiency of weight sharing with the robust search capabilities of evolution.