Weight Sharing (NAS)

The foundation of weight sharing is the supernet, an over-parameterized graph that embeds all possible operations and connections defined by the search space. During a single training run, the supernet learns a shared set of weights that act as a proxy for the performance of its countless sub-networks. This is distinct from training each candidate architecture from scratch.

To evaluate a specific architecture, the NAS algorithm samples a path through the supernet, activating only the corresponding operations and connections. The performance of this subnetwork is estimated using the shared weights, bypassing the need for retraining. Common strategies include:

• Uniform sampling for initial supernet training.
• Policy-guided sampling (e.g., via reinforcement learning or gradients) to focus on promising regions.
• In-place evaluation where accuracy on a validation set is measured without updating weights.

In Differentiable Architecture Search (DARTS), weight sharing enables gradient-based optimization. Alongside the shared network weights, continuous architecture parameters (alphas) are introduced for each candidate operation. The bi-level optimization process jointly learns:

• Shared Weights via standard gradient descent on training data.
• Architecture Parameters via gradient descent on validation data. The final discrete architecture is derived by selecting the operation with the highest alpha value at each choice point.

The primary challenge is ranking consistency: the performance ranking of subnetworks using shared weights must correlate highly with their ranking if trained independently. Poor correlation leads to suboptimal search results. Factors affecting this include:

• Supernet training stability and optimization schedule.
• Interference between vastly different subnetworks sharing weights.
• Capacity of the supernet to represent diverse architectures well. Techniques like path dropout and sandwich rule training are used to improve consistency.

Weight sharing enables the One-Shot NAS paradigm, where the supernet is trained only once. After training, the search process becomes a low-cost evaluation of sampled architectures. Once-For-All (OFA) extends this by training a supernet to support diverse subnet configurations (depth, width, kernel size) from which many hardware-tailored models can be extracted instantly for different latency or memory constraints without retraining.

For Hardware-Aware NAS, weight sharing is critical for evaluating hardware metrics (latency, energy) efficiently. Instead of deploying each candidate on real hardware—a slow process—a pre-built hardware cost model (e.g., a latency lookup table) is queried using the sampled subnetwork's configuration. This allows the search to optimize the joint objective of accuracy and efficiency with orders-of-magnitude less time and compute than non-weight-sharing methods.

A supernet is an over-parameterized neural network that embeds all possible operations and pathways defined by a NAS search space. It serves as the foundational structure for weight sharing.

• Mechanism: During one-shot NAS training, the supernet's shared weights are optimized. Sub-architectures (child models) inherit these weights for evaluation without independent training.
• Analogy: Think of it as a 'motherboard' where different 'expansion card' configurations (sub-networks) can be plugged in and tested using the same underlying power and data buses.
• Key Challenge: Balancing the co-adaptation of operations within the supernet to ensure fair ranking of sub-networks.

One-Shot NAS is a family of methods where a single supernet is trained once. The performance of myriad sub-architectures is then estimated by evaluating them as different configurations of this shared-weight network.

• Core Principle: Decouples architecture search from weight training, reducing search cost from thousands of GPU days to a few.
• Process: 1) Train the supernet (often with path dropout or other techniques). 2) Search for high-performing sub-architectures using the frozen supernet weights. 3) Optionally retrain the best-found architecture from scratch.
• Examples: DARTS (Differentiable ARchiTecture Search) and ProxylessNAS are prominent one-shot methods, though they differ in how the search is conducted.

In NAS, a performance estimator is any method used to predict the final accuracy or hardware efficiency of a candidate architecture without completing its full, standalone training cycle.

• Weight Sharing as an Estimator: The validation accuracy of a sub-network using shared supernet weights is a low-cost performance estimator, though it can be biased.
• Zero-Cost Proxies: Even faster estimators that use metrics like gradient norms or synaptic flow calculated from a single forward/backward pass on a small data batch. Examples include NASWOT and SynFlow.
• Hardware Cost Models: These estimators predict latency, memory, or energy usage, often using pre-characterized lookup tables or small neural network predictors.

The search space defines the universe of all possible neural network architectures a NAS algorithm can explore. Weight sharing operates within this constrained design space.

• Components: Typically parameterized by:
  • Operator Choices: Convolution type (standard, depthwise), pooling, skip connection, or zero operation (i.e., removal).
  • Connectivity: Which layers connect to which others.
  • Hyperparameters: Number of channels, kernel sizes, number of layers.
• Impact on Weight Sharing: A well-designed search space is crucial. If it's too large or contains poorly interacting operations, weight sharing becomes ineffective as the supernet cannot learn universally good representations for all sub-networks.

Once-For-All is a seminal one-shot NAS and training paradigm that explicitly leverages weight sharing for efficient deployment across diverse hardware platforms.

• Methodology: A single supernet is progressively trained to support a large number of sub-networks varying in depth, width, kernel size, and resolution.
• Deployment: After training, specialized sub-networks for different latency or memory constraints (e.g., a 50ms phone vs. a 10ms microcontroller) can be extracted instantly without retraining.
• Significance: OFA demonstrated the practical utility of weight sharing for neural hardware co-design, enabling a single training process to yield a family of models tailored to various edge devices.

DARTS is a gradient-based one-shot NAS method that makes the search space continuous and optimizes architecture parameters via gradient descent, relying fundamentally on weight sharing.

• Mechanism: It introduces continuous relaxation: instead of choosing a single operation (e.g., conv3x3) between two nodes, it uses a weighted sum of all possible operations. The mixing weights (architecture parameters) are learned alongside the supernet weights.
• Weight Sharing Role: The supernet's shared weights are optimized concurrently with these architecture parameters. The discrete final architecture is derived by selecting the operation with the highest learned weight for each connection.
• Trade-off: While highly efficient, DARTS can suffer from performance estimation error due to the discrepancy between the continuous optimization and the final discrete architecture.