Differentiable Architecture Search (DARTS)

DARTS replaces the discrete choice between candidate operations (e.g., 3x3 convolution, 5x5 convolution, skip connection) with a continuous mixture. Each edge in the computation cell is represented as a weighted sum of all possible operations. The architecture parameters (alpha) control these mixture weights. This relaxation transforms the search from a combinatorial problem into a continuous one solvable via gradient descent.

The search process is formalized as a bilevel optimization problem. Two sets of parameters are optimized simultaneously:

• Model weights (w): Optimized on the training data via standard gradient descent to minimize training loss.
• Architecture parameters (α): Optimized on the validation data, with the gradient computed through the optimized model weights. The objective is to find architecture parameters α that minimize the validation loss, where the model weights w are themselves a function of α obtained by minimizing the training loss.

By leveraging the continuous relaxation, DARTS uses gradient descent to optimize architecture parameters, a drastic efficiency improvement over reinforcement learning or evolutionary-based NAS methods. It approximates the gradient ∇αL_val(w*, α) using a first-order or second-order approximation, avoiding the prohibitive cost of fully optimizing w for every gradient step on α. This reduces search time from thousands of GPU days to orders of magnitude less.

DARTS searches for a repeating computational cell (normal and reduction cells) rather than a full network. The final architecture is derived by:

1. Retaining the operations with the highest learned architecture parameter (α) weights on each edge.
2. Stacking the discovered cells to form the final network. This discretization step converts the continuous, over-parameterized supernet into a discrete, efficient architecture for final training from scratch.

All candidate architectures within the search space are subgraphs of a single over-parameterized supernet. Their performance is estimated by evaluating them as different sub-networks of this shared supernet, a technique known as weight sharing. This avoids the need to train each candidate architecture independently from scratch, which is the primary computational bottleneck in traditional NAS methods.

While groundbreaking, DARTS has known limitations that spurred significant follow-up research:

• Performance Collapse: The continuous relaxation can lead to a dominance of skip-connect operations, causing performance degradation. Methods like DARTS+ and PC-DARTS add early stopping or regularization to mitigate this.
• Optimization Gap: The performance ranking of architectures in the supernet (search) may not correlate perfectly with their performance when trained in isolation (evaluation).
• Memory Overhead: The supernet must hold all candidate operation parameters simultaneously, leading to high GPU memory consumption.

Neural Architecture Search (NAS) is a subfield of automated machine learning (AutoML) focused on algorithmically discovering high-performing neural network architectures for a given dataset and task, rather than relying on human-designed structures. NAS methods are broadly categorized by their search strategy:

• Reinforcement Learning-based: A controller agent (e.g., an RNN) proposes architectures and is rewarded based on their trained performance.
• Evolutionary: A population of architectures is evolved using selection, mutation, and crossover operations.
• Gradient-based (like DARTS): The search space is relaxed into a continuous, differentiable form, allowing architecture parameters to be optimized via gradient descent. The primary challenge is the immense computational cost, as evaluating a single architecture requires full training.

One-Shot Neural Architecture Search (NAS) is a family of efficient methods where the entire search space is encoded within a single, over-parameterized supernet. The core technique enabling this efficiency is weight sharing.

How it works:

• A supernet is constructed containing every possible operation (e.g., convolution, pooling, skip connection) in the search space.
• Candidate architectures are represented as subgraphs of this supernet and inherit the pre-trained weights of the corresponding supernet components.
• This allows for the evaluation of thousands of architectures without training each from scratch, as their performance is estimated using the shared weights. DARTS is a prominent one-shot method that uses a continuous relaxation of the supernet for gradient-based optimization of the architectural parameters.

In NAS and hyperparameter optimization, the search space is the formally defined set of all possible configurations an automated algorithm can explore. For DARTS and architecture search, this defines the building blocks and connectivity of potential networks.

Key components of a NAS search space:

• Cell-based design: The search focuses on discovering a optimal computational "cell" (normal and reduction cells), which is then stacked to form the final network. This is the design used in DARTS.
• Operation choices: For each edge or node in the cell, the algorithm selects from a predefined set of operations (e.g., 3x3 separable conv, 5x5 dilated conv, max pooling, identity, zero).
• Macro-architecture: Fixed outer skeleton parameters, such as the number of cells, initial channels, and downsampling layers. A well-designed search space balances expressiveness (ability to find high-performing models) with tractability (size the search algorithm can handle).

Gradient-based optimization in the context of AutoML refers to methods that treat hyperparameters or architectural choices as differentiable parameters, enabling the use of efficient gradient descent. DARTS is the seminal example for architecture search.

Mechanism:

1. Continuous Relaxation: The discrete choice between operations (e.g., conv or pool) is relaxed using a softmax over continuous architecture parameters (alpha).
2. Bilevel Optimization: The problem is framed as optimizing architecture parameters (alpha) on a validation set, while the model weights (w) are optimized on a training set. This creates a nested optimization loop.
3. Gradient Computation: The gradient of the validation loss with respect to alpha is computed, often using an approximation (e.g., one-step unrolled gradient), to update the architectural parameters. This approach contrasts with black-box methods like reinforcement learning or evolution, offering superior search efficiency but introducing challenges like memory overhead and performance discretization gap.

Hardware-Aware Neural Architecture Search (NAS) is an optimization paradigm that incorporates target deployment constraints directly into the search objective. Instead of solely maximizing accuracy, it seeks architectures that optimize a Pareto front of trade-offs between metrics like:

• Latency (inference time)
• Memory footprint
• Energy consumption
• Computational FLOPs

Implementation: The search loss function is often augmented with a penalty term for the hardware metric (e.g., Loss = CrossEntropy + λ * Latency). Specialized techniques include:

• Using a latency lookup table or a pre-trained surrogate model to predict hardware performance without on-device measurement for every candidate.
• Searching directly on the target device (e.g., a mobile phone) for the most accurate measurements. This is critical for deploying models on edge devices, smartphones, or specialized neural processing units (NPUs) where resources are constrained.

Zero-cost proxies are metrics used to estimate the performance of a neural network architecture without requiring any training. They are designed for extreme efficiency in large-scale architecture screening or as part of a multi-stage NAS pipeline.

How they work: These proxies compute a score based on properties of the network at initialization or from a single forward/backward pass on a mini-batch of data. Common examples include:

• grad_norm: The norm of gradients after one backward pass.
• snip: Sensitivity of the loss to pruning each parameter at initialization.
• synflow: A measure of weight synergy that avoids layer-wise scaling issues.
• jacob_cov: The covariance of Jacobians, related to the network's expressivity.

While not as accurate as full training, zero-cost proxies can correlate with final trained accuracy, allowing for the rapid pruning of thousands of poor architectures, thereby drastically reducing the search cost before applying more expensive methods like DARTS.