Differentiable NAS (DARTS)

DARTS replaces the discrete choice between candidate operations (e.g., 3x3 convolution, 5x5 convolution, max pooling) with a continuous, weighted mixture. Each edge in the architecture supergraph is represented by a set of architecture parameters (alpha) that act as mixing coefficients for a softmax over all possible operations. This relaxation makes the search space differentiable, enabling the use of gradient descent to jointly optimize the model weights and the architecture parameters.

The search process is framed as a bi-level optimization problem. The goal is to find architecture parameters (alpha) that minimize the validation loss, where the model weights (w) are obtained by minimizing the training loss.

• Inner Loop: Update model weights w on the training split via standard gradient descent.
• Outer Loop: Update architecture parameters alpha on the validation split, using gradients that backpropagate through the weight optimization step. This separation enforces that the selected architecture generalizes to unseen validation data.

After the continuous optimization converges, a discrete architecture must be extracted from the soft mixture. For each node in the computational graph, the strongest operations are selected:

• On each edge, the operation with the highest learned alpha value is retained.
• For normal and reduction cells, the top-k strongest incoming edges to each intermediate node are kept, where k is a hyperparameter (typically 2). This discretization step converts the continuous supernet into a practical, efficient network for final training and evaluation.

DARTS dramatically reduces the computational cost of Neural Architecture Search (NAS).

• Traditional NAS (e.g., using Reinforcement Learning or Evolutionary Algorithms): Requires training thousands of candidate architectures from scratch to completion, often consuming tens of thousands of GPU hours.
• DARTS: Trains a single, over-parameterized supernet once. The search is completed in the order of 1-4 GPU days on datasets like CIFAR-10, as architecture selection is performed via efficient gradient computation rather than exhaustive evaluation.

DARTS operates on a cell-based search space to ensure scalability. Instead of searching an entire macro-architecture, it searches for two types of reusable, modular cells:

• Normal Cell: Preserves spatial resolution.
• Reduction Cell: Reduces spatial resolution (typically via stride 2). The discovered cells are then stacked repeatedly to form the final convolutional network. This design constrains the search to a manageable space and allows the discovered architecture to generalize to different dataset sizes by simply changing the number of cell repeats.

Despite its breakthrough in efficiency, DARTS has known limitations that spurred further research:

• Performance Collapse with Deep Supernets: The continuous relaxation can lead to a rich-get-richer effect where certain operations (often skip connections or null ops) are over-selected, degrading final performance, especially when searching for many epochs.
• High GPU Memory Footprint: The supernet must keep all candidate operations active in memory during search, which can be prohibitive.
• Optimization Gap: The optimal architecture parameters alpha in the relaxed space may not correspond to the optimal discrete architecture after pruning, a discrepancy known as the discretization gap. Methods like DARTS+ and RobustDARTS were proposed to address these stability issues.

Neural Architecture Search (NAS) is the overarching field of automating the design of neural network architectures. It treats the network's topology—layer types, connections, hyperparameters—as a searchable space. NAS algorithms evaluate candidate architectures to discover high-performing models for a specific task and dataset.

• Core Goal: Automate the manual, expert-driven process of model design.
• Search Methods: Includes reinforcement learning (e.g., training a controller RNN), evolutionary algorithms, and gradient-based methods like DARTS.
• Trade-off: Balances search efficiency (compute cost) against the final performance of the discovered architecture.
• Example: Google's pioneering work used RL to design NASNet, a convolutional cell that outperformed hand-designed models on ImageNet.

Conditional computation is a paradigm where a model dynamically activates different subsets of its parameters or computational pathways based on the input. This enables efficient, input-specific processing rather than applying the full, fixed network to every sample.

• Mechanism: A routing function (e.g., a gating network) decides which parts of the model to use.
• Benefit: Enables massive model capacity (many parameters) with sub-linear computational cost per example.
• Key Architectures: Mixture of Experts (MoE) is a prime example, where a gating network routes tokens to specialized sub-networks.
• Contrast with DARTS: DARTS searches for a single, static architecture. Conditional computation defines a dynamic architecture that changes per input.

A Mixture of Experts (MoE) layer consists of multiple expert networks (e.g., feed-forward layers) and a gating network. For each input, the gating network produces a sparse combination of experts, activating only a few. This allows for dramatically increasing model capacity without a proportional increase in computation.

• Sparsity: Critical for efficiency. Only the top-k experts (e.g., top-2) are activated per token.
• Scale: Foundation of trillion-parameter models like Google's Switch Transformers and Mixtral 8x7B.
• Connection to DARTS: While DARTS searches for connections, MoE defines a dynamic, conditional routing structure. Advanced NAS can be used to search the design of the experts or the gating mechanism itself.

HyperNetworks are neural networks that generate the weights for a separate, primary network. Instead of learning static weights, the primary model's parameters are dynamically produced as a function of some input, task embedding, or latent code.

• Dynamic Parameterization: Enables a single model to encapsulate a family of behaviors or adapt instantly to new tasks.
• Use Cases: Few-shot learning, continual learning, and input-specific modulation (e.g., dynamic filter networks in vision).
• Relation to NAS/DARTS: HyperNetworks can be seen as learning a continuous space of architectures defined by their generated weights. DARTS learns a continuous relaxation of architectural choices (edges, ops) within a fixed supernet.

Meta-learning, or 'learning to learn,' trains a model on a distribution of tasks so it can rapidly adapt to new, unseen tasks with minimal data. The model internalizes a prior over task structures, allowing for fast adaptation via a few gradient steps or a forward pass.

• Goal: Generalize the learning process itself, not just a single function.
• Algorithms: Include MAML (Model-Agnostic Meta-Learning), which optimizes for good initial parameters, and metric-based approaches like prototypical networks.
• Connection to DARTS: Meta-learning can be applied to architecture search. DARTS can be viewed as a form of meta-learning where the task is 'find a good architecture,' and the adaptation is done via gradient descent on the validation set. The learned architectural parameters represent a prior over good model structures for a data domain.

One-Shot NAS is a family of NAS methods that construct a single, over-parameterized supernetwork (or one-shot model) that encompasses all possible architectures in the search space. Architectures are evaluated as subgraphs of this supernet by sharing its weights, eliminating the need to train each candidate from scratch.

• Weight Sharing: The supernet is trained once. Candidate architectures inherit weights from the corresponding paths in the supernet.
• Efficiency: Dramatically reduces search cost from thousands of GPU days to a few.
• DARTS as One-Shot NAS: DARTS is a specific, differentiable implementation of the one-shot paradigm. It trains the supernet with continuous architectural parameters and uses gradient descent to solve the search problem, whereas other one-shot methods may use evolutionary search or reinforcement learning on the frozen supernet.