Evolutionary NAS

Evolutionary NAS maintains a population of candidate architectures, unlike sequential methods. This allows for parallel exploration of diverse regions of the search space. The population evolves over generations, where each member (an architecture) is evaluated for fitness (e.g., validation accuracy).

• Key Advantage: Inherent parallelism and diversity help avoid local minima.
• Core Mechanism: A selection pressure (e.g., tournament selection) promotes high-fitness architectures to the next generation.

New architectures are created by applying variation operators to the selected population.

• Mutation: Randomly modifies an existing architecture. Examples include:
  • Adding or removing a layer.
  • Changing a layer's operation (e.g., swapping a 3x3 convolution for a 5x5 convolution).
  • Altering connectivity (e.g., adding a skip connection).
• Crossover (Recombination): Combines components from two parent architectures to produce an offspring, mixing their traits.

These operators enable exploration of novel architectures while exploiting successful building blocks.

The driving force of evolution is the fitness function, which quantifies architecture quality. It's typically the model's performance on a validation set after training. To manage computational cost, strategies include:

• Lower-fidelity estimates: Training for fewer epochs or on a data subset.
• Weight sharing: Using a supernet where child architectures inherit weights, avoiding training from scratch.

After evaluation, a survivor selection strategy (e.g., replacing the worst individuals or using elitism) determines which architectures persist, ensuring the population's average fitness improves over generations.

Practical deployment requires balancing accuracy with computational constraints. Evolutionary NAS excels at multi-objective optimization, searching for architectures on the Pareto front.

Common objectives include:

• Model Accuracy (Primary task performance).
• Inference Latency (Hardware-specific speed).
• Parameter Count / Model Size (Memory footprint).
• Energy Consumption (Power efficiency).

The algorithm evolves architectures that represent optimal trade-offs, allowing engineers to select a model based on deployment priorities.

The search space defines all possible architectures the algorithm can generate. In Evolutionary NAS, it's often represented as:

• Cell-based search spaces: The algorithm evolves repeatable normal and reduction cells, which are stacked to form the full network. This promotes transferable, scalable designs.
• Graph-based representations: An architecture is encoded as a computational graph where nodes are feature maps and edges are operations (conv, pool, etc.). Mutation and crossover act directly on this graph structure.

This flexible representation allows the evolution of complex, non-sequential topologies with branching and skip connections.

Evolutionary NAS is often contrasted with gradient-based methods like DARTS. Key differentiators:

• Black-Box Optimization: Does not require the search space to be differentiable; can handle discrete, non-differentiable operations and constraints (e.g., hardware latency).
• Global Search: The population mechanism can escape local optima more effectively than gradient descent in a relaxed space.
• Computational Cost: Historically more expensive, but modern improvements like weight sharing have narrowed the gap.
• Resulting Architectures: Often produces irregular, highly optimized designs that might be less intuitive than human-designed patterns.

Neural Architecture Search (NAS) is the overarching subfield of AutoML focused on algorithmically discovering high-performing neural network architectures for a given dataset and task. It automates the design process that was traditionally manual.

• Core Goal: Find an architecture that maximizes a performance metric (e.g., accuracy) under constraints (e.g., model size, latency).
• Search Components: Defined by a search space (possible operations like convolutions), a search strategy (e.g., evolutionary, reinforcement learning, gradient-based), and a performance estimation strategy (how to evaluate candidates).
• Evolutionary NAS is one prominent search strategy within this field.

Hyperparameter Optimization (HPO) is the automated process of searching for the optimal set of hyperparameters that govern a model's training process. While NAS searches for architecture, HPO tunes the knobs of a given architecture.

• Examples: Learning rate, batch size, optimizer settings, regularization strength.
• Key Methods: Include Bayesian Optimization, Random Search, Grid Search, and Population-Based Training (PBT).
• Relationship to NAS: NAS can be viewed as a higher-level form of HPO where the hyperparameters define the model's graph structure. Many NAS methods, including evolutionary approaches, use HPO techniques within their inner loops.

Differentiable Architecture Search (DARTS) is a gradient-based alternative to evolutionary NAS. It formulates the search space as a continuous relaxation, allowing architecture parameters to be optimized via standard gradient descent alongside model weights.

• Core Mechanism: Represents the choice between operations (e.g., 3x3 conv, 5x5 conv, skip connection) as a softmax over continuous weights. The search becomes a bi-level optimization problem.
• Advantage: Significantly more computationally efficient than many reinforcement learning or evolutionary methods, often completing a search in a few GPU days.
• Trade-off: The continuous relaxation is an approximation, and the final discretized architecture may not perform exactly as predicted during the search.

Hardware-Aware NAS is an optimization paradigm that incorporates target deployment constraints directly into the architecture search objective. Evolutionary NAS is frequently used in this context.

• Objective: Discover architectures that maximize accuracy while satisfying constraints like inference latency, memory footprint, energy consumption, or model size.
• Multi-Objective Search: The search aims to find architectures on the Pareto front, representing the optimal trade-offs between accuracy and the hardware metric.
• Practical Impact: Essential for deploying models on edge devices, mobile phones, or specialized accelerators (NPUs/TPUs) where resources are strictly limited.

One-Shot NAS is a family of efficient methods where the entire search space is encoded within a single, over-parameterized supernet. Weight sharing is the key technique that enables this efficiency.

• Process: A supernet containing all possible operations is trained once. Candidate architectures are evaluated as subgraphs of this supernet, inheriting (sharing) the pre-trained weights without needing independent training.
• Benefit: Reduces search cost from thousands of GPU days to potentially a single day.
• Challenge: The performance ranking of subnets using shared weights is only a proxy for their performance if trained in isolation (rank correlation problem). Evolutionary algorithms can be used to search over this one-shot model.

Population-Based Training (PBT) is a hybrid optimization algorithm that combines aspects of evolutionary strategies and traditional hyperparameter tuning. It simultaneously trains a population of models.

• Mechanism: Poorly performing models in the population can exploit by copying the weights from better performers. All models then explore by randomly perturbing their hyperparameters.
• Difference from Evolutionary NAS: PBT primarily evolves hyperparameters and model weights for a fixed architecture. Evolutionary NAS evolves the architecture itself. However, the concepts of population, selection, and mutation are shared.
• Use Case: Efficiently optimizing learning schedules and hyperparameters during a single training run.