Weight Sharing

At the heart of weight sharing is the supernet (or one-shot model), a single, densely connected neural network that embeds all possible candidate architectures within its structure. Instead of training each candidate from scratch, sub-networks (or paths) inherit and share parameters from this supernet.

• Architecture as a Subgraph: Each candidate is a specific subgraph of the supernet's computational graph.
• Weight Inheritance: The weights for operations (e.g., a 3x3 convolution) are shared across all candidate architectures that include that operation.
• Benefit: This reduces the cost of evaluating a candidate from full training to a simple forward/backward pass on the shared weights, enabling the search over vast spaces (e.g., >10¹³ architectures) with feasible compute.

To train the supernet and evaluate architectures, specific paths must be activated. This is managed through path sampling or a continuous relaxation of the architecture choice.

• Discrete Sampling: Methods like ENAS (Efficient NAS) use a controller to sample a single path per training batch, updating only the weights along that path.
• Differentiable Relaxation: Methods like DARTS (Differentiable Architecture Search) relax the discrete choice between operations (e.g., conv3x3, conv5x5, skip-connect) into a continuous, weighted sum. Architecture parameters (alphas) are learned via gradient descent alongside network weights (thetas).
• Key Distinction: Sampling is more memory-efficient but introduces variance; relaxation enables gradient-based optimization but requires significant memory to maintain all parallel paths.

Weight sharing creates a bilevel optimization problem. The goal is to find the optimal architecture, but its performance is measured by the weights it inherits from the supernet, which are themselves being optimized.

The two intertwined objectives are:

1. Inner Loop (Weight Optimization): Minimize the training loss to find good shared weights (W) for the supernet.
2. Outer Loop (Architecture Optimization): Maximize the validation performance of sub-networks using the shared weights (W) to find the optimal architecture (α).

In practice, this is often approximated by alternating optimization: training network weights on a training split and updating architecture parameters on a validation split.

A core assumption of weight sharing is ranking consistency: the performance ranking of candidate architectures using shared weights should correlate highly with their ranking if trained independently. Violation of this assumption leads to performance collapse, where the search selects poor architectures.

Causes include:

• Supernet Overfitting: The supernet memorizes the training data, making shared weights poor proxies for final performance.
• Operation Fairness: Certain operations (e.g., skip connections) may receive more gradient updates during supernet training, biasing the search.
• Mitigation Strategies: Techniques like path dropout, sandwich rule training (training both the largest and smallest sub-networks each step), and fairness regularization are used to maintain ranking fidelity.

Weight sharing is a substrate that can be combined with various search strategies to select the final architecture.

• Evolutionary Search: After supernet training, evolutionary algorithms sample and mutate sub-networks, evaluating them via a fast forward pass on shared weights.
• Gradient-Based Search: As used in DARTS, architecture parameters are directly optimized via gradient descent after the relaxation.
• Reinforcement Learning: An RL controller samples paths; the reward is the performance of the path using shared weights (or after a short fine-tuning step).
• Random Search with Weight Sharing: Even simple random search becomes powerful, as evaluating thousands of random candidates is computationally trivial once the supernet is trained.

The output of a weight-sharing NAS process is a set of continuous architecture parameters (e.g., alphas in DARTS) or a discrete champion architecture. Discretization is the process of deriving the final, deployable network.

• Operation Selection: For each node in the computational graph, the operation with the highest learned architecture weight is selected.
• Standalone Retraining: The final, discrete architecture is retrained from scratch (without weight sharing) on the full dataset. This is critical, as the shared weights are an approximation. The discovered architecture typically outperforms hand-designed networks when trained properly.
• Performance Gain: The true value of weight-sharing NAS is not the shared weights themselves, but the discovered, efficient topology that is then fully optimized.

A primary challenge is ensuring the performance ranking of sub-networks (architectures) within the supernet accurately reflects their stand-alone performance. Pathological coupling can occur, where the shared optimization favors certain paths, causing the supernet's rankings to be unreliable. This is often mitigated by techniques like uniform path sampling and sandwich rule training to ensure all candidate architectures receive equitable optimization.

Training a single, massively over-parameterized network containing all candidate architectures is inherently unstable. The optimization trajectory is a complex compromise between thousands of sub-models, leading to:

• Gradient conflict between different paths.
• High optimization variance and sensitivity to hyperparameters like learning rate.
• Risk of the supernet converging to a poor local optimum that biases the search. Techniques like progressive shrinking (ENAS) and architecture dropout are used to stabilize training.

Weight sharing methods like Differentiable Architecture Search (DARTS) perform search in a continuous, relaxed space where architectural choices are represented by soft, differentiable weights. The discretization gap refers to the performance drop when these soft weights are converted (discretized) into a final, hard architecture for standalone training. A large gap indicates the supernet's evaluation was not predictive, often addressed by post-search retraining or architecture refinement.

Weight sharing imposes strict requirements on the NAS search space. The supernet must be a directed acyclic graph (DAG) where all candidate architectures are subgraphs. This constrains flexibility:

• Operations must be layer-wise (e.g., choice of 3x3 or 5x5 conv at a specific stage).
• Macro-architectural changes (e.g., novel connectivity patterns like dense blocks) are harder to encode.
• The search space size grows combinatorially, risking supernet overfitting to the search validation set.

While far cheaper than training each candidate independently, weight sharing is not free. The supernet is larger than any single candidate, requiring significant GPU memory to hold all potential operation weights simultaneously. Training involves:

• Activating multiple paths per batch, increasing forward/backward pass complexity.
• Managing large, sparse gradient tensors.
• Techniques like memory-efficient supernets and gradient checkpointing are essential for scaling to large search spaces like those for vision transformers.

The ultimate test is whether the discovered architecture, when trained from scratch without weight sharing, achieves its promised performance. Factors threatening generalization include:

• Supernet overfitting to the search procedure's specific hyperparameters and data splits.
• Operation bias where the supernet's shared Batch Normalization statistics may not transfer.
• Validation requires a rigorous hold-out dataset not used during the search phase to provide an unbiased estimate of final task performance.

Neural Architecture Search (NAS) is the automated process of discovering high-performing neural network architectures for a specific dataset and task. Instead of relying on human intuition, NAS algorithms explore a vast search space of possible layer types, connections, and hyperparameters.

• Core Goal: Automate the design of model architectures, which is often more complex than tuning hyperparameters.
• Key Methods: Include reinforcement learning, evolutionary algorithms (Evolutionary NAS), and gradient-based approaches like Differentiable Architecture Search (DARTS).
• Challenge: The primary bottleneck is the immense computational cost of training and evaluating thousands of candidate architectures, which weight sharing directly addresses.

One-Shot Neural Architecture Search (NAS) is a family of methods where the entire architectural search space is encoded within a single, over-parameterized supernet. All candidate architectures are subgraphs of this supernet and share its weights.

• Mechanism: The supernet is trained once. Candidate architectures are then evaluated by inheriting the shared weights, requiring no independent training.
• Efficiency: This reduces the cost of architecture evaluation from thousands of GPU days to a matter of days or hours.
• Relation to Weight Sharing: One-Shot NAS is the primary paradigm that relies entirely on weight sharing. Techniques like DARTS and ENAS (Efficient NAS) are prominent examples.

A supernet (or one-shot model) is a single, large, over-parameterized neural network that encompasses all possible architectures within a defined NAS search space. It is the foundational construct that enables weight sharing.

• Structure: Contains every possible operation (e.g., 3x3 convolution, 5x5 convolution, skip connection) at every potential layer position.
• Training: The supernet is trained with techniques like path dropout or sandwich rule sampling to ensure all sub-architectures learn usable weights.
• Purpose: After training, any candidate architecture can be instantiated by selecting a specific path through the supernet and using the corresponding shared weights for immediate performance estimation.

Differentiable Architecture Search (DARTS) is a gradient-based NAS method that makes the architecture search space continuously differentiable. It relaxes the discrete choice of operations into a soft, weighted mixture, allowing optimization via gradient descent.

• How it works: Introduces a set of continuous architecture parameters (alphas) that weight the contribution of each operation (e.g., conv, pool) at each edge in the supernet.
• Bi-level Optimization: The model weights are optimized on the training set, while the architecture parameters are optimized on a validation set.
• Weight Sharing: DARTS is a quintessential One-Shot NAS method. The supernet's shared weights are optimized concurrently with the architecture parameters, making search highly efficient.

In NAS and hyperparameter optimization, the search space is the formally defined universe of all possible configurations an automated algorithm can explore. For NAS, this defines the building blocks and rules for constructing neural networks.

• Components: Typically includes choices of operation types (convolution, pooling, attention), number of layers, filter sizes, activation functions, and connection patterns (e.g., residual, dense).
• Design Impact: A well-designed search space balances expressiveness (ability to find good models) with size (tractability for search).
• Constraint for Weight Sharing: The supernet must be designed to represent this entire search space. The feasibility and efficiency of weight sharing are directly determined by how the search space is formulated.

Hardware-Aware Neural Architecture Search (NAS) extends the standard NAS objective (e.g., validation accuracy) to include deployment constraints like latency, memory footprint, or energy consumption. The goal is to find architectures on the Pareto front of optimal accuracy-efficiency trade-offs.

• Multi-Objective Search: The search algorithm optimizes for both accuracy and a hardware metric (e.g., FLOPs, measured inference time on a target device).
• Integration with Weight Sharing: Efficiency metrics can be estimated for sub-architectures within a supernet without full deployment, allowing hardware-aware search to be performed efficiently via weight-sharing paradigms like One-Shot NAS.
• Practical Outcome: Discovers models tailored for specific platforms, such as mobile phones (low latency) or microcontrollers (TinyML).