Evolutionary NAS

Evolutionary NAS maintains a population of candidate neural network architectures, analogous to a gene pool. This population is iteratively updated, allowing the algorithm to explore diverse regions of the search space simultaneously and maintain multiple promising solutions, rather than following a single gradient path. This diversity helps avoid premature convergence to suboptimal architectures.

New architectures are generated using biologically inspired genetic operators:

• Mutation: Randomly modifies an existing architecture (e.g., changing a layer's operation from a 3x3 convolution to a 5x5 depthwise convolution).
• Crossover (Recombination): Creates a child architecture by combining components from two parent architectures (e.g., taking the first half of one network and the second half of another). These operations introduce controlled randomness and enable the exploration of novel architectural patterns.

Each architecture in the population is assigned a fitness score, typically based on its accuracy on a validation set and its hardware efficiency (e.g., latency, memory footprint). A selection pressure is applied where architectures with higher fitness scores are more likely to be chosen as 'parents' for the next generation. This survival of the fittest mechanism steadily drives the population toward higher-performing designs.

For TinyML and edge deployment, the fitness function is multi-objective, directly optimizing for hardware constraints. The algorithm evolves architectures that balance accuracy with critical metrics:

• Peak Memory Usage (RAM/Flash)
• Inference Latency
• Energy Consumption This transforms the search from purely accuracy-driven to a hardware-in-the-loop co-design process, ensuring discovered models are viable for microcontrollers.

The search proceeds in discrete generations. Each generation involves evaluating the current population, selecting parents, applying genetic operators to produce offspring, and forming a new population. Performance improves incrementally over hundreds or thousands of generations. This process is inherently parallelizable, as multiple architectures can be trained and evaluated concurrently.

Evolutionary NAS is particularly well-suited for hardware-aware search on microcontrollers due to several key strengths:

• Black-Box Optimization: It does not require differentiable search spaces, allowing exploration of discrete, complex operations common in efficient networks.
• Global Search: The population-based approach is less likely to get stuck in local optima compared to pure gradient methods.
• Pareto Frontier Discovery: Naturally discovers a diverse set of architectures representing optimal trade-offs (the Pareto frontier) between accuracy, latency, and size.

Evolutionary NAS can directly incorporate hardware-in-the-loop profiling into its fitness function. Instead of relying solely on proxy metrics or simulated latency, candidate architectures are compiled and benchmarked on the actual target microcontroller (e.g., an Arm Cortex-M4). The measured latency (ms), peak RAM usage (KB), and energy consumption (µJ/inference) become the primary selection criteria, ensuring discovered models are pragmatically deployable.

The search explicitly optimizes for the Pareto frontier across competing objectives critical for TinyML:

• Model Accuracy (e.g., classification F1-score)
• Flash Memory Footprint (often < 500 KB)
• Inference Latency (e.g., < 100 ms for real-time sensor processing)
• Energy per Inference (targeting microjoule range) Evolutionary algorithms maintain a diverse population, naturally discovering a suite of architectures that represent optimal trade-offs, such as a tiny 50 KB model for always-on sensing versus a larger 200 KB model for periodic high-accuracy tasks.

The evolutionary search operates within a TinyML-optimized search space that excludes operations prohibitive on MCUs. This space is defined by:

• Efficient Base Operators: Depthwise-separable convolutions, pointwise convolutions, and linear layers.
• Excluded Operators: Standard convolutions with large kernels, batch normalization (often fused or replaced with group normalization), and complex activations.
• Constraint-Aware Parameters: Searching over channel widths (e.g., 8, 16, 32) and layer depths that keep tensor dimensions within SRAM limits. The mutation and crossover operations are designed to only produce valid, compile-ready architectures for the target hardware.

Evolutionary NAS can be conducted with quantization-aware training (QAT) throughout the search process. Candidate architectures are trained and evaluated using simulated integer arithmetic (e.g., 8-bit weights and activations). This ensures the final evolved model is inherently robust to the precision loss from post-training quantization, a mandatory step for MCU deployment. This avoids the common pitfall where a high-accuracy floating-point model collapses in performance after being quantized.

A practical application is searching for a keyword spotting (KWS) model to run on a battery-powered IoT device. The evolutionary search might start with a population of small convolutional neural networks. The fitness function evaluates each candidate on:

1. Accuracy on the Google Speech Commands dataset.
2. Measured latency on an ESP32 microcontroller.
3. Peak SRAM usage during inference. Through successive generations, the algorithm evolves architectures that maximize accuracy while guaranteeing < 20 ms latency and < 50 KB of RAM, resulting in a model suitable for always-listening applications.

For hardware-aware search under extreme constraints, evolutionary NAS offers distinct benefits:

• Handles Non-Differentiable Objectives: It can optimize directly for measured hardware metrics (latency, energy) which are non-differentiable and thus challenging for gradient-based methods like DARTS.
• Global Search: Less prone to getting stuck in local minima of the architecture space, which is crucial when the optimal TinyML model may be structurally very different from standard designs.
• Constraint Satisfaction: Easy to implement hard constraints (e.g., 'model must use < 256 KB of flash') by simply discarding architectures that violate them during selection.

Neural Architecture Search (NAS) is the overarching automated machine learning technique for discovering optimal neural network architectures. It replaces manual, expert-driven design with an algorithmic search process. Evolutionary NAS is one specific search strategy within this field.

• Core Concept: A NAS algorithm explores a predefined search space of possible architectures, guided by a search strategy, and uses a performance estimator to evaluate candidates.
• Objective: To find an architecture that maximizes a given objective, typically validation accuracy on a target task and dataset.

The search strategy is the core algorithm that governs how a NAS system explores the search space. Evolutionary NAS is one prominent strategy, alongside others like reinforcement learning and gradient-based methods.

• Evolutionary Algorithms: Use principles of biological evolution (selection, mutation, crossover) on a population of architectures.
• Reinforcement Learning (RL-NAS): A controller network is trained to generate architectures, receiving a reward based on their performance.
• Gradient-Based (e.g., DARTS): Relaxes the discrete search space to be continuous, allowing optimization via gradient descent.
• Random Search: Serves as a simple but important baseline for comparison.

Hardware-Aware NAS is a critical variant that incorporates deployment constraints directly into the search objective. Instead of just accuracy, it optimizes for metrics like latency, memory use, and energy consumption on specific hardware.

• Key Constraints: Latency constraint, memory constraint, and energy budget.
• Hardware Cost Model: A predictive function (e.g., lookup table, neural network) that estimates the on-device performance of a candidate architecture without needing full deployment for every evaluation.
• Goal: Discovers Pareto-optimal architectures that balance accuracy with real-world efficiency, essential for edge and TinyML deployment.

Multi-Objective NAS explicitly optimizes for several, often competing, objectives simultaneously. Evolutionary algorithms are naturally suited for this, as they can maintain a diverse population representing different trade-offs.

• Common Objectives: Model accuracy, inference latency, model size (parameters), memory footprint, and energy consumption.
• Pareto Frontier Search: The goal is to discover the Pareto frontier—a set of architectures where improving one objective necessitates worsening another. This provides engineers with a suite of optimal choices.
• Trade-off Analysis: Essential for practical deployment where resources are constrained, moving beyond pure accuracy maximization.

MCU-NAS refers to Hardware-Aware NAS methods specifically tailored for the extreme constraints of microcontrollers. It represents the direct application of evolutionary and other NAS strategies to the TinyML domain.

• Extreme Constraints: Optimizes for kilobyte-scale RAM/Flash, microjoule-level energy budgets, and MHz-range clock speeds.
• Specialized Metrics: May include peak memory usage, execution time measured in milliseconds, and compatibility with fixed-point arithmetic.
• Co-Design: Often involves tight integration with microcontroller inference optimization techniques like quantization and pruning during the search (e.g., Quantization-Aware NAS).

Once-For-All (OFA) is an efficient NAS and training paradigm that leverages a supernet and weight sharing. While not exclusively evolutionary, its principles are often combined with evolutionary search for sub-network selection.

• Supernet: A single, over-parameterized neural network that embeds all possible architectures in the search space.
• Weight Sharing: Sub-architectures (subnets) inherit weights from the supernet, allowing their performance to be estimated without costly independent training.
• Evolutionary Search Role: After the supernet is trained, an evolutionary algorithm can efficiently search through millions of subnets for those best meeting hardware constraints, as evaluating a candidate only requires inference, not training.