Neural Architecture Search (NAS)

The search space defines the universe of possible neural network architectures the NAS algorithm can explore. It is a critical design choice that balances flexibility with tractability.

• Macro vs. Micro Search: Macro search spaces define entire chain-structured networks (e.g., number of layers, types of layers). Micro search spaces define the structure of repeated computational cells (e.g., the operations within a cell for a CNN), which are then stacked to form the full network.
• Common Primitives: Typical operations include convolutions (3x3, 5x5), pooling (max, average), identity (skip connections), and zero (dropping a connection).
• Constraint Engineering: The search space is often constrained by hardware considerations (e.g., FLOPs, latency) or architectural principles (e.g., enforcing residual connections) to ensure discovered models are practical.

This is the algorithm that explores the search space to find high-performing architectures. The choice of strategy dictates the efficiency and effectiveness of the NAS process.

• Reinforcement Learning (RL): Uses an RNN controller to generate architecture descriptions (actions) and receives the model's accuracy (reward) to update its policy. Pioneered by Zoph & Le (2017).
• Evolutionary Algorithms: Treats architectures as individuals in a population. Uses mutation and crossover to generate new candidates, selecting the fittest based on performance.
• Differentiable Architecture Search (DARTS): Formulates the search as a continuous optimization problem. A supernet contains all possible operations, weighted by architecture parameters (alphas). The search jointly trains network weights and alphas via gradient descent, then discretizes the final architecture by selecting the operation with the highest alpha.
• Bayesian Optimization & Random Search: Often used as strong baselines, demonstrating that a well-defined search space can sometimes be effectively navigated with simpler, less computationally intensive methods.

Evaluating a candidate architecture's performance is the most computationally expensive part of NAS. Estimation strategies aim to predict final accuracy without full training.

• Lower Fidelity Estimates: Training for fewer epochs, on a subset of data, or with a smaller model (fewer channels, cells) provides a cheap, correlated proxy for final performance.
• Weight Sharing / One-Shot NAS: All candidate architectures are treated as subgraphs of a single, over-parameterized supernet. The supernet's weights are trained once. Candidate architectures are evaluated by inheriting the corresponding weights from the supernet without independent training, enabling orders-of-magnitude faster search.
• Learning Curve Extrapolation: Predicting the final validation accuracy based on the early learning trajectory of a partially trained model.
• Surrogate Models: Training a predictor (e.g., a neural network) to map an architecture's representation directly to its estimated performance, amortizing the cost after initial data collection.

The primary historical barrier to NAS has been its extreme computational cost, often requiring thousands of GPU days. Modern research focuses on drastically reducing this cost.

• The Cost Problem: Early RL-based NAS (e.g., NASNet) required over 2000 GPU days. This made NAS inaccessible to most researchers and organizations.
• Efficiency Drivers: The shift to weight-sharing (one-shot) methods and differentiable search (DARTS) reduced search costs to the order of 1-4 GPU days, making NAS practical.
• Proxy Tasks: Searching on a smaller dataset (e.g., CIFAR-10) or with a smaller network backbone before transferring the discovered cell structure to a larger dataset (e.g., ImageNet).
• Hardware-in-the-Loop Search: Integrating real hardware latency or energy consumption metrics directly into the search objective to find architectures that are not just accurate but also deployable on target devices.

NAS is rarely just about finding a topology. Modern approaches often jointly search for the architecture and other critical hyperparameters.

• Hyperparameter Integration: The search space can be expanded to include choices like learning rate schedules, optimizer types, data augmentation policies, and activation functions. This blurs the line between NAS and AutoML.
• Neural Architecture & Quantization Search (NAQS): Simultaneously searches for the model architecture and the optimal bit-width for quantizing each layer, optimizing for accuracy under hardware constraints.
• Search for Robust Architectures: The objective can be modified to search for architectures that are not only accurate but also robust to adversarial examples or common corruptions, integrating adversarial training or data augmentation into the search loop.

A core premise of NAS is that an architecture discovered for one task or dataset should possess inductive biases that generalize.

• Cell-Based Transfer: The standard practice of discovering a convolutional or attention-based cell on a small proxy task (CIFAR-10) and stacking it to form a network for a large-scale task (ImageNet, COCO). This demonstrates the transferability of fundamental building blocks.
• Task-Agnostic Search: Searching for general-purpose architectures that perform well across a diverse set of tasks (e.g., image classification, object detection, semantic segmentation). This tests the universality of the discovered inductive bias.
• Zero-Shot Proxies: Research into metrics that can predict architecture quality without any training (e.g., based on gradient norms, neural tangent kernel properties). While not yet fully reliable, they represent the ultimate goal for low-cost search generalization.

NAS employs three primary algorithmic families to explore the architecture space:

• Reinforcement Learning (RL): A controller network (RNN) proposes child architectures, which are trained and evaluated. The controller's policy is updated via policy gradient methods (e.g., REINFORCE) based on the child's validation performance.
• Evolutionary Algorithms: A population of architectures is maintained. High-performing 'parent' architectures are selected and mutated (e.g., altering a layer type, filter size) to produce 'offspring' for the next generation.
• Differentiable NAS (DARTS): The search space is relaxed into a continuous supergraph where the choice between operations is represented as a softmax over continuous architecture weights. The optimal discrete architecture is derived by selecting the operation with the highest learned weight.

The search space defines the set of all possible architectures the algorithm can explore. Key design paradigms include:

• Cell-based Search: The algorithm searches for optimal computation cells (normal and reduction cells) which are then stacked to form the full network. This enables transferability across datasets and scales (e.g., NASNet, AmoebaNet).
• Hierarchical Search: The space is structured at multiple levels, from primitive operations to macro-architectural motifs, allowing for more structured and human-interpretable discoveries.
• Weight-Sharing (One-Shot NAS): All candidate architectures are treated as subgraphs of a single, over-parameterized supernet. This allows for efficient evaluation by sharing weights, drastically reducing search cost from thousands of GPU days to a few.

Evaluating every candidate architecture by full training is computationally prohibitive. NAS uses proxy methods:

• Lower-Fidelity Estimation: Training on a subset of data, for fewer epochs, or with a smaller model size.
• Weight Inheritance / Network Morphism: New architectures inherit weights from a parent network, enabling warm-start training.
• One-Shot / Supernet-Based: The performance of a sub-architecture is estimated by its performance as part of the shared-weight supernet, often via a validation pass without additional training. The correlation between this proxy score and final performance is critical for success.

NAS has produced architectures that rival or surpass human-designed counterparts:

• NASNet & AmoebaNet: Discovered via RL and evolution, these cell-based architectures achieved state-of-the-art on ImageNet and became standard backbones.
• EfficientNet: Uses NAS to perform a compound scaling optimization, jointly scaling network depth, width, and resolution for optimal efficiency/accuracy trade-off.
• DARTS Cells: The differentiable search method discovered efficient convolutional cells that are highly effective in downstream tasks.
• Transformer Variants: NAS has been applied to discover more efficient transformer architectures for NLP and vision, optimizing attention patterns, feed-forward layers, and normalization placement.

Modern NAS optimizes not just for accuracy but for deployment constraints, making it crucial for edge AI and production systems.

• Multi-Objective Search: The search optimizes a Pareto frontier balancing metrics like accuracy, latency (on a target device), FLOPs, and model size.
• Platform-Specific Search: The search incorporates a latency predictor—a small model trained to estimate inference time on specific hardware (e.g., a mobile CPU, Google Edge TPU)—directly into the optimization loop. This yields models like MobileNetV3 and MNasNet.

Despite its power, NAS faces significant challenges driving current research:

• Computational Cost: Even with weight-sharing, searches require substantial resources, raising barriers to entry and environmental concerns.
• Generalization & Robustness: Architectures found on one dataset/task may not transfer optimally. Research focuses on task-agnostic NAS and searching for robust architectures.
• Joint Search with Hyperparameters & Training Recipes: The frontier is end-to-end AutoML, where NAS jointly optimizes the architecture, optimizer, learning rate schedule, and data augmentation policies.
• Theory & Reproducibility: The stochastic nature of searches and sensitivity to hyperparameters can make results difficult to reproduce. Efforts aim to better understand the dynamics of architecture optimization.

Conditional computation is a broad paradigm where a neural network dynamically activates different subsets of its parameters or computational pathways based on the specific input. This is a core principle behind architectures discovered by NAS, which often learn to create specialized data-dependent flows. Key implementations include:

• Mixture of Experts (MoE): Routes inputs to different expert sub-networks.
• Dynamic Filter Networks: Generates convolutional kernels on-the-fly.
• Adaptive Computation Time (ACT): Allows RNNs to perform a variable number of computational steps per input.

The goal is to improve model capacity and efficiency by avoiding the cost of evaluating the entire, dense network for every sample, a property NAS algorithms frequently optimize for.

A Mixture of Experts (MoE) is a neural network architecture composed of multiple specialized sub-networks (experts) and a gating network that dynamically routes each input to the most relevant subset of experts. This is a prime example of conditional computation and a common output structure for NAS in large language models. Sparse MoE variants, which activate only a small, fixed number of experts (e.g., top-2) per token, are crucial for scalability:

• Enables Massive Capacity: Models can have trillions of parameters while keeping computational cost per token manageable.
• NAS Application: Search algorithms can optimize the number of experts, their sizes, and the gating mechanism.
• Parallelism Challenge: Requires expert parallelism strategies for distributed training across multiple GPUs.

HyperNetworks are neural networks that generate the weights for a separate, primary network. This decouples the model's architecture from its static parameterization, allowing for dynamic, input-conditional, or task-specific weight generation. In the context of NAS and dynamic architectures:

• Architecture Encoding: A hypernetwork can generate weights for a novel architecture discovered by a NAS controller, enabling rapid weight prediction without full training from scratch.
• Meta-Learning Link: HyperNetworks are a tool for fast adaptation, as a meta-learned hypernetwork can produce weights for a new task based on a few examples.
• Efficiency Trade-off: The overhead of running the hypernetwork must be justified by the benefits of dynamic parameterization or reduced training time.

Automated Adaptation refers to the use of AutoML techniques—including NAS, hyperparameter optimization (HPO), and meta-learning—to automatically adjust a machine learning system to changing data distributions (concept drift) or new tasks without manual intervention. This is a higher-level automation goal that encompasses NAS:

• End-to-End Pipeline: AutoML for adaptation manages the entire cycle: detecting drift, triggering a search for a new architecture or hyperparameters, and validating the updated model.
• Beyond Static Search: Unlike one-off NAS for a fixed dataset, automated adaptation implies a continuous, production-integrated process.
• Key for Continuous Learning: It enables systems within the Continuous Model Learning pillar to autonomously evolve their model structures in response to feedback and new data streams.