Weight Sharing

A supernet is an over-parameterized, directed acyclic graph that embeds all possible candidate architectures within a predefined search space. It contains every possible layer, connection, and operation. During a one-shot training phase, a single set of shared weights is trained across this entire graph. This allows any sub-network (or child model) to inherit a performance estimate without being trained from scratch, reducing evaluation cost from thousands of GPU days to mere hours.

To evaluate a specific architecture, a path (or sub-graph) is sampled from the supernet. Only the operations and connections along this path are activated for a forward/backward pass. The shared weights corresponding to this path are used. Key methods include:

• Uniform Sampling: Randomly sampling paths to train the supernet fairly.
• Gradient-Based Sampling: Using a controller or architecture parameters to bias sampling toward promising regions. Performance metrics (accuracy, latency) for the sub-network are estimated using these shared weights, providing a proxy for its stand-alone performance.

Differentiable Architecture Search (DARTS) formulates NAS as a bi-level optimization problem where both architecture parameters (α) and shared model weights (w) are optimized simultaneously via gradient descent.

• The search space is relaxed into a continuous mixture of operations.
• The architecture parameters control the contribution of each operation (e.g., conv3x3, sep_conv5x5).
• After joint training, a discrete architecture is derived by selecting the operation with the highest α value for each edge. This method makes weight sharing efficient by replacing expensive reinforcement learning or evolutionary searches with gradient-based optimization.

A central challenge is weight entanglement, where the optimal weights for one sub-network interfere with those of another, reducing the accuracy of performance estimation. This occurs due to:

• Co-adaptation: Weights become specialized to the most frequently sampled paths.
• Forgetting: Weights for rarely sampled paths become stale. Techniques to mitigate this include:
• Path Dropout: Randomly dropping operations during training.
• Fairness Regularization: Encouraging uniform path sampling.
• Sandwich Rule: Training both the largest and smallest sub-networks in each batch.

The performance estimation gap is the discrepancy between a sub-network's accuracy when evaluated with shared supernet weights versus when it is retrained from scratch in isolation. This gap is the primary metric for weight sharing efficacy. A small gap (< 1%) indicates high-fidelity sharing. Factors widening the gap include:

• Severe weight entanglement.
• Insufficient supernet training.
• Mismatch between supernet optimization and stand-alone training recipes. Advanced techniques like progressive shrinking (OFA) and in-place distillation are used to minimize this gap.

The Once-For-All (OFA) network is a supernet designed to support diverse depth, width, kernel size, and resolution configurations. It uses progressive shrinking as its core training algorithm:

1. Train the largest full network.
2. Finetune this network to support smaller sub-networks by progressively adding them to the training distribution.
3. This elastic training ensures that the shared weights perform well across all supported configurations, not just the largest. This allows a single OFA model to be deployed across vastly different hardware constraints (from cloud to edge) without retraining.

The efficacy of weight sharing hinges on how the supernet is optimized. Poor strategies lead to unreliable ranking of sub-architectures.

• Uniform Sampling: Each training step samples a random sub-architecture uniformly from the search space. Simple but can under-optimize complex sub-networks.
• FairNAS / SPOS (Single Path One-Shot): Employs stricter fairness constraints, ensuring each operation or path is sampled an equal number of times during training to reduce bias.
• Progressive Shrinking (OFA): Starts with the largest sub-network and progressively introduces smaller ones, stabilizing training.
• Sandwich Rule: Each training batch includes the largest model, the smallest model, and several random models, improving weight generalizability across scales.

Weight sharing introduces specific optimization and evaluation artifacts that must be managed.

• Optimization Bias (Ranking Disorder): The performance ranking of sub-architectures under shared weights may not correlate with their ranking if trained in isolation. This undermines the search's validity.
• Multi-Model Forgetting: The supernet's shared weights struggle to simultaneously maintain high performance for vastly different sub-architectures, akin to catastrophic forgetting in continual learning.
• Depth/Gradient Bias: Deeper or wider sub-networks within the supernet may receive stronger gradients during training, unfairly advantaging them in the subsequent evaluation phase.
• Mitigation: Advanced sampling strategies, regularization, and path dropout are used to encourage more equitable weight optimization.

This approach abstracts weight sharing to a higher level. Instead of a static supernet, a hypernetwork—a small neural network—is trained to generate the weights for a main network based on an architecture description. The hypernetwork's parameters are the shared entity.

• Mechanism: An architecture encoding (e.g., a graph) is input to the hypernetwork, which outputs the corresponding weights for the main network.
• Benefit: Decouples weight generation from the main network's structure, enabling efficient learning of weights for a continuous space of architectures, not just a discrete set defined by a supernet.
• Application: Particularly useful for conditional computation and generating weights for novel, unseen architectures at inference time.

Neural architecture search (NAS) is the automated process of discovering high-performing neural network architectures for a given task and dataset. It treats the network design—choices like layer types, connections, and hyperparameters—as an optimization problem.

• Core Objective: To find architectures that maximize a performance metric (e.g., accuracy) while potentially minimizing computational cost (e.g., latency, parameters).
• Key Methods: Include reinforcement learning, evolutionary algorithms, and gradient-based approaches like Differentiable NAS (DNAS).
• Role of Weight Sharing: NAS algorithms often employ weight sharing within a supernet to make the search process computationally feasible by avoiding training thousands of candidate models from scratch.

One-shot neural architecture search is an efficient NAS paradigm where a single, over-parameterized supernet is trained once. All candidate sub-architectures (subnets) inherit and share these weights for evaluation.

• Mechanism: The supernet encompasses all possible operations and connections in the search space. After its one-shot training, subnets are assessed by simply activating their specific computational paths without independent training.
• Primary Benefit: Drastically reduces search cost from thousands of GPU days to a few, as it eliminates the need to train every candidate model to convergence.
• Direct Link: This is the primary context where weight sharing is implemented, as the supernet's parameters are the shared resource for all sub-architectures.

Differentiable neural architecture search (DNAS) formulates architecture search as a continuous optimization problem. It relaxes the discrete search space by representing architectural choices with continuous architecture parameters, which are optimized via gradient descent alongside the model weights.

• Supernet as a Differentiable Graph: The supernet is constructed so that the choice between operations (e.g., 3x3 conv vs. 5x5 conv) is modeled as a softmax over continuous variables.
• Joint Optimization: The shared network weights and the architecture parameters are optimized simultaneously on the training data.
• Connection to Weight Sharing: DNAS is a specific, gradient-based instantiation of the one-shot NAS paradigm, relying entirely on a weight-shared supernet for efficient search.

A supernet (or one-shot model) is an over-parameterized neural network that contains all possible architectural components defined within a NAS search space. It is the central vehicle for weight sharing.

• Structure: It is essentially a directed acyclic graph where each edge or node represents a choice between different operations (convolution, pooling, identity, etc.).
• Function: During one-shot training, all paths are trained jointly. A specific subnet is defined by sampling a single path or subset of operations through this graph.
• Critical Property: The supernet must be designed to enable fair assessment of its subnets; poorly designed supernet training can lead to inaccurate ranking of candidate architectures.

In neural architecture search, the search space is the predefined set of all possible neural network architectures that the automated algorithm can explore. It defines the building blocks and rules for how they can be connected.

• Components: Typically includes choices for layer operations (convolution type, kernel size), number of channels (width), number of layers (depth), and skip connection patterns.
• Design Impact: A well-designed search space balances expressiveness (ability to find high-performing models) with size (to make the search tractable).
• Relation to Weight Sharing: The supernet is a parameterized instantiation of the entire search space. Weight sharing occurs across all the architectural variants contained within this defined space.

Hardware-aware neural architecture search is a specialized form of NAS that directly incorporates hardware performance metrics into the search objective. It aims to find architectures optimized for specific deployment targets like mobile phones, embedded devices, or specific accelerators (NPUs/GPUs).

• Optimization Targets: Metrics include inference latency, memory footprint, energy consumption, and model size.
• Integration: Uses lookup tables, latency predictors, or on-device measurements to estimate the hardware cost of a candidate architecture during the search.
• Role of Weight Sharing: It relies on efficient search methods like one-shot NAS with weight sharing to practically explore the vast space of architectures while evaluating each against costly-to-measure hardware constraints.