Supernet

A supernet is an over-parameterized neural network that embeds the entire NAS search space within a single, large graph. It contains all possible operations (e.g., 3x3 conv, 5x5 depthwise conv, skip connection) and pathways defined by the search space. This design allows it to function as a superset of all candidate architectures, with each sub-network represented by a specific architectural configuration (alpha) that activates a subset of the supernet's paths and operations.

The core efficiency mechanism of a supernet is weight sharing. Instead of training each candidate architecture from scratch, all sub-networks inherit and share weights from the single, jointly trained supernet. This allows for the approximate evaluation of a candidate's performance by simply activating its specific path and reading the shared weights.

• Key Benefit: Reduces search cost from thousands of GPU-days to a few.
• Critical Challenge: Requires careful training to mitigate interference and co-adaptation between disparate sub-networks, ensuring shared weights provide a fair performance ranking.

In methods like DARTS (Differentiable Architecture Search), the supernet enables a continuous relaxation of the search space. Discrete choices between operations are softened using a softmax over architecture parameters (alpha). This allows the relative importance of each operation to be optimized via gradient descent alongside the network weights.

• The supernet is trained on two sets of parameters: network weights (w) and architecture parameters (alpha).
• After joint optimization, the final discrete architecture is derived by selecting the operation with the highest alpha value at each choice point.

Advanced supernets, such as those in the Once-For-All (OFA) network, are designed with elastic dimensions. This allows a single trained supernet to be dynamically tailored to diverse hardware constraints by adjusting:

• Depth: Number of layers.
• Width: Number of channels per layer.
• Kernel Size: Spatial dimensions of convolutional filters.
• Resolution: Input image size.

This elasticity enables the extraction of a Pareto-optimal frontier of sub-networks optimized for different trade-offs (e.g., accuracy vs. latency) from one supernet, without retraining.

The supernet acts as a performance predictor. By sharing weights, it provides a rapid, in-situ method to estimate the accuracy of any sub-architecture within its search space. This estimation is performed via validation on a hold-out set, where only the forward pass for the specific sub-graph is executed.

• This is far faster than full independent training but is an approximation.
• The quality of this ranking is paramount and is the primary metric for evaluating supernet training strategies. Poor weight sharing leads to ranking disorder, where the supernet's performance estimates do not correlate with the sub-network's stand-alone performance.

For Hardware-Aware NAS, the supernet is the vehicle for integrating hardware feedback into the search loop. A hardware cost model (e.g., a latency lookup table) can be queried during the supernet's architectural sampling or gradient step.

• The search objective becomes a multi-objective function (e.g., accuracy + latency).
• The supernet allows for efficient exploration of the trade-off surface, identifying sub-networks that meet strict memory, latency, or energy constraints for target microcontrollers or edge accelerators.

Neural Architecture Search (NAS) is an automated machine learning technique that discovers optimal neural network architectures for a given task and dataset, replacing manual design with algorithmic search. The process involves three core components:

• Search Space: Defines the set of all possible architectures.
• Search Strategy: The algorithm (e.g., RL, evolution, gradient-based) that explores the space.
• Performance Estimator: Evaluates candidate architectures, often via weight sharing or proxy metrics. NAS is the overarching paradigm that supernets were created to accelerate.

One-Shot NAS is a family of neural architecture search methods that train a single, over-parameterized supernet once. All possible sub-architectures (child models) share the weights of this parent network. This enables the evaluation of thousands of candidates via simple weight sharing, without the need to retrain each one from scratch. It dramatically reduces search cost from thousands of GPU days to a few. Supernets are the central engine of One-Shot NAS methodologies.

Weight sharing is the critical technique that makes supernet-based NAS efficient. Instead of training each candidate architecture independently, all sub-netpaths within the supernet share a common set of parameters. When a specific path (architecture) is activated during search, it uses a subset of these shared weights. This allows for the performance of a candidate to be estimated by evaluating it as a sub-graph of the already-trained supernet, bypassing costly standalone training. The quality of these shared weights directly determines the ranking accuracy of the search.

Differentiable Architecture Search (DARTS) is a seminal gradient-based NAS method that formulates the supernet's search space as a continuous, differentiable optimization problem. It introduces architecture parameters (alphas) as continuous variables that represent the probability of selecting between operations (e.g., conv3x3, sep_conv5x5). The supernet is trained with standard gradient descent on both network weights and these alphas. After training, a discrete architecture is derived by retaining the operations with the highest alpha values. DARTS popularized the differentiable supernet paradigm.

Once-For-All (OFA) is a comprehensive one-shot NAS and training paradigm designed for edge device diversity. It trains a single large supernet to support an enormous number of sub-networks varying in depth, width, and kernel size. After training, specialized models for different hardware constraints (e.g., a 50ms phone vs. a 10ms microcontroller) can be extracted without any retraining. OFA demonstrates the practical deployment power of supernets, enabling a single training run to produce a factory of efficient, ready-to-use models.

The search space is the set of all possible neural network architectures that a NAS algorithm can propose. It defines the building blocks and rules for connectivity. For a supernet, the search space is explicitly encoded in its structure. Common dimensions include:

• Operator Search: Choice of layer type (convolution, pooling, identity, zero).
• Channel Search: Number of output filters per layer.
• Kernel Size Search: Spatial dimensions of convolutional filters.
• Connectivity Search: Which layers are connected (e.g., in a cell-based or macro search). The supernet is an over-parameterized instantiation of this entire search space.