Neural Architecture Search (NAS)

The search space defines the universe of all possible neural network architectures the NAS algorithm can consider. For TinyML, this space is heavily constrained by hardware targets.

• Macro-architectural choices: Number of layers, types of layers (convolution, depthwise convolution, fully connected), and connection patterns (e.g., residual blocks, inverted residuals).
• Micro-architectural parameters: Kernel size (3x3, 5x5), number of filters per layer, expansion ratios for mobile networks, and activation functions.
• Hardware-aware constraints: The space is often limited to operations and topologies known to be efficient on microcontrollers, such as avoiding large fully connected layers and favoring depthwise separable convolutions.

The search strategy is the algorithm that navigates the search space to discover high-performing architectures. It balances exploration of new designs with exploitation of promising regions.

• Reinforcement Learning (RL): Uses an RNN controller to generate architecture descriptions, which are then trained and rewarded based on validation accuracy.
• Evolutionary Algorithms: Applies genetic operations (mutation, crossover) to a population of architectures, selecting the fittest for the next generation.
• Gradient-Based Methods: Treats the architecture selection as a continuous optimization problem (e.g., using Differentiable Architecture Search (DARTS)), allowing efficient search via gradient descent.
• Bayesian Optimization: Models the relationship between architecture and performance as a probabilistic surrogate function to guide the search.

The performance estimation strategy is the method for evaluating a candidate architecture's quality (e.g., accuracy, latency) without the prohibitive cost of fully training every candidate from scratch.

• Low-Fidelity Estimation: Training candidates for fewer epochs or on a smaller dataset.
• Weight Sharing / One-Shot Models: Training a single, over-parameterized supernet (like a Once-For-All network) that shares weights across all candidate subnetworks. Performance is estimated by evaluating different subgraphs of this supernet.
• Predictor-Based: Training a surrogate model (e.g., a neural network or regression model) to predict the final performance of an architecture based on its metadata or a learned embedding.
• Hardware-in-the-Loop Profiling: Directly measuring true latency, memory usage, and power consumption by deploying and profiling the candidate on the target microcontroller or a cycle-accurate simulator.

For TinyML, the NAS objective function extends beyond validation accuracy to include critical hardware deployment metrics. The algorithm searches for architectures that optimize a multi-objective cost function.

• Primary Objectives: Model accuracy on the target task.
• Hardware Constraints: Peak RAM usage, flash storage footprint, inference latency (in milliseconds), and energy consumption (in millijoules per inference).
• Multi-Objective Optimization: Often formulated as finding architectures on the Pareto frontier, the set of designs where no single metric (e.g., accuracy) can be improved without degrading another (e.g., latency).
• Example Objective: Maximize(Accuracy) subject to: Latency < 50ms, PeakRAM < 256KB.

Unlike traditional NAS focused solely on accuracy, TinyML NAS incorporates hardware-specific metrics as primary search objectives. The search algorithm is guided by a multi-objective reward function that penalizes architectures exceeding target constraints.

Key search targets include:

• Peak RAM/Flash Usage: Models must fit within the microcontroller's limited SRAM (often < 512KB) and flash memory (often < 1MB).
• Inference Latency: Measured in milliseconds, targeting real-time sensor processing (e.g., < 100ms for keyword spotting).
• Energy Consumption: Estimated in millijoules per inference, critical for battery-powered devices.
• Multiply-Accumulate (MAC) Operations: A proxy for compute cost, directly correlated with latency and energy.

Search spaces are constrained from the start, excluding operations like large dense layers or standard convolutions that are prohibitive on MCUs.

The set of possible operations and connections (the search space) is meticulously crafted for microcontroller efficiency. It heavily favors depthwise separable convolutions, which drastically reduce parameters and MACs compared to standard convolutions. Common building blocks include:

• MobileNet-style inverted residuals: Efficient blocks with expansion and projection layers.
• Squeeze-and-Excitation (SE) attention: Lightweight channel-wise gating to boost accuracy with minimal cost.
• Grouped & pointwise convolutions: To further reduce computational complexity.
• Activation functions: Choices like ReLU6 (clipped ReLU) are preferred for simpler quantization.
• Skip connections: To facilitate gradient flow in very deep, efficient networks.

The search space explicitly excludes operations with high memory or compute overhead, such as batch normalization during inference (folded into weights) or large kernel sizes.

Training a unique model for every device variant is infeasible. The Once-For-All (OFA) approach trains a single, large supernet encompassing many possible subnetworks of different depths, widths, and kernel sizes. After this one-time training, specialized submodels for specific latency or memory targets can be extracted without retraining.

This is ideal for TinyML because:

• Manufacturing Variance: A single supernet can yield models for MCUs with slightly different clock speeds or memory.
• Dynamic Voltage & Frequency Scaling (DVFS): A device can extract a smaller submodel when on battery power and a larger one when plugged in.
• Product Families: One supernet serves an entire product line, from a basic to a premium sensor.

The supernet is trained with progressive shrinking, gradually introducing smaller subnetworks into the training process to maintain the accuracy of all contained architectures.

Differentiable Architecture Search (DARTS) is a popular NAS method adapted for TinyML. Instead of evaluating thousands of discrete architectures, it relaxes the search space to be continuous. A architecture parameter (alpha) is associated with each possible operation (e.g., 3x3 conv, 5x5 depthwise conv, skip connection, zero).

During the search phase, the model is a mixture of all operations, weighted by their alpha parameters. The search optimizes two sets of parameters simultaneously:

1. The standard network weights (using gradient descent).
2. The architecture parameters, alpha (using gradient descent).

After search, a discrete architecture is derived by retaining only the operation with the highest alpha value at each choice point. This method is significantly more compute-efficient than reinforcement learning or evolutionary-based NAS, making it more accessible for edge-focused research.

The most advanced TinyML NAS frameworks perform joint architecture and quantization policy search. The search algorithm doesn't just choose operations; it also decides the bit-width (precision) for weights and activations for each layer.

For example, the search might learn that:

• The first convolutional layer benefits from 8-bit weights and activations for robustness.
• Intermediate depthwise layers can be reduced to 4-bit with minimal accuracy loss.
• The final classification layer requires 8-bit activations.

This results in heterogeneously quantized models where the precision varies per layer, achieving an optimal accuracy-to-efficiency trade-off. The search reward function directly incorporates the cost of mixed-precision arithmetic on the target hardware.

A canonical TinyML NAS success is designing models for always-on keyword spotting (e.g., "Hey Google," "Alexa") on microcontrollers. The constraints are extreme: ~250KB RAM, ~50ms latency, and microwatts of power.

NAS-discovered architectures for this task, such as variants of BC-ResNet or TC-ResNet, typically feature:

• 1D temporal convolutions instead of 2D spectrogram processing.
• Carefully tuned dilation rates to capture long-range audio dependencies without increasing kernel size.
• Extremely narrow channel widths in early layers, expanding only where necessary.

These NAS-generated models consistently outperform hand-designed baselines like DSCNN, achieving >96% accuracy on the Google Speech Commands dataset while meeting all hardware constraints. They demonstrate NAS's ability to find non-intuitive, highly efficient patterns that human designers might overlook.

A specialized form of NAS where the search algorithm optimizes the neural network architecture not only for task accuracy but also for specific hardware deployment constraints. The search space and reward function are designed to directly target metrics like:

• Latency (inference time on the target MCU)
• Memory footprint (SRAM and Flash usage)
• Power consumption (energy per inference)
• Compute operations (MAC count)

This process often involves building a latency/energy lookup table or using a performance predictor model for the target hardware to evaluate candidate architectures without full deployment.

A supernet training methodology that creates a large, weight-shared network encompassing many possible subnetworks of varying depths, widths, and kernel sizes. Trained once with progressive shrinking and distillation techniques, it allows for the efficient extraction of numerous specialized submodels tailored to different resource constraints without retraining. This is highly synergistic with NAS for TinyML, as it provides a pre-searched and trained space of efficient architectures from which to select the optimal subnetwork for a given microcontroller's profile.

The overarching family of techniques used to reduce a neural network's size, computational cost, and energy consumption for deployment. NAS often produces architectures that are inherently efficient, but they are frequently combined with post-search compression. Key techniques include:

• Quantization: Reducing numerical precision of weights/activations (e.g., to INT8).
• Pruning: Removing redundant parameters (weights, neurons, filters).
• Knowledge Distillation: Training a small student model to mimic a larger teacher.

NAS and compression are complementary: NAS designs the efficient blueprint, and compression further optimizes the implementation of that blueprint.

The defined universe of all possible neural network architectures that a NAS algorithm can explore. For TinyML, this space is carefully constrained to be hardware-feasible. It includes discrete choices over:

• Layer types (Conv2D, DepthwiseConv, Fully Connected)
• Operation parameters (kernel size, stride, number of filters)
• Connectivity patterns (skip connections, branching)
• Macro-architecture (number of stages, resolution scaling)

Designing the search space is critical—it must be expressive enough to find high-performing models but small enough to search efficiently. Common spaces include cell-based (searching for a repeating block) and macro-architecture search.

The algorithm that navigates the search space to discover high-performing architectures. Strategies balance exploration and evaluation cost. Major categories include:

• Reinforcement Learning (RL): Uses an RNN controller trained with policy gradients to generate architectures.
• Evolutionary Algorithms: Applies mutation and crossover to a population of architectures, selecting the fittest.
• Differentiable Architecture Search (DARTS): Relaxes the search space to be continuous, allowing gradient-based optimization of architecture parameters.
• Bayesian Optimization: Uses a surrogate model to predict architecture performance and guide the search. For resource-constrained search, weight-sharing (evaluating many architectures within a single supernet) is a key efficiency technique.

The method used to evaluate the quality (accuracy, latency) of a candidate architecture during the NAS process without fully training it from scratch, which is computationally prohibitive. Strategies include:

• Low-fidelity estimation: Training for fewer epochs or on a subset of data.
• Weight sharing / One-shot NAS: Training a single supernet; candidate architectures are evaluated as its subnetworks.
• Learning curve extrapolation: Predicting final performance from early training epochs.
• Hardware-in-the-loop profiling: Directly measuring latency/power on the target device or an accurate simulator. The choice of estimation strategy dramatically impacts the total computational cost and practical feasibility of the NAS process.