Neural Architecture Search (NAS)

The search space defines the universe of possible neural network architectures the NAS algorithm can explore. For TinyML, this space is heavily constrained by hardware targets.

• Macro-architectures: Define overall network topology (e.g., number of layers, layer types, connection patterns).
• Micro-architectures: Define operations within a cell or layer (e.g., choice of 3x3 depthwise convolution, 5x5 separable convolution, identity, or zero operation).
• Hardware-aware constraints: Limit depth, width, kernel sizes, and activation functions based on target microcontroller memory (SRAM/Flash) and latency budgets.

The search strategy is the optimization algorithm that navigates the search space to find high-performing architectures. It balances exploration and computational cost.

• Reinforcement Learning (RL): Uses an RNN controller to generate architecture descriptions, which are trained and rewarded based on validation accuracy.
• Evolutionary Algorithms: Treats architectures as a population, using mutation and crossover to evolve better designs over generations.
• Differentiable Architecture Search (DARTS): Relaxes the discrete search space to be continuous, allowing efficient gradient-based optimization. This is a one-shot method where a supernet is trained, and the final architecture is derived from learned architectural weights.
• Bayesian Optimization: Models the performance landscape to predict promising architectures with fewer evaluations.

The performance estimation strategy evaluates candidate architectures. Accurate estimation is critical, as fully training each candidate is computationally prohibitive.

• Low-fidelity estimators: Train on a subset of data, for fewer epochs, or with lower resolution.
• Weight sharing / One-shot models: Train a single over-parameterized supernetwork that shares weights across all candidate sub-architectures. Candidate performance is estimated by inheriting weights from the supernet without standalone training.
• Learning curve extrapolation: Predicts final performance from early training epochs.
• Hardware-in-the-loop measurement: Directly profiles latency, memory usage, and energy consumption on the target microcontroller or an accurate simulator/emulator.

Hardware-aware cost metrics are integrated into the NAS objective function to directly optimize for deployment constraints, not just accuracy.

• Multi-objective optimization: Balances accuracy against metrics like:
  • Latency (inference time on target MCU)
  • Peak memory usage (SRAM for activations)
  • Model size (Flash for parameters)
  • Energy consumption (mJ per inference)
• Proxy models: Use fast, pre-computed models (e.g., based on MAC count, memory access patterns) to estimate hardware cost without full deployment for every candidate.
• Pareto-optimal frontiers: NAS outputs a set of architectures representing the best trade-offs between accuracy and each hardware constraint.

TinyNAS refers to NAS systems specifically designed for microcontroller-level constraints, often producing MCUNet-style models.

Key adaptations include:

• Extremely constrained search spaces aligned with MCU ISA (e.g., favoring INT8 operations, avoiding large kernels).
• On-device measurement as the primary performance estimator via cycle-accurate emulators.
• Joint search over neural architecture and compiler optimizations (e.g., schedule, tiling).
• Focus on peak SRAM usage as the primary bottleneck, often more critical than FLOPs. Example: MCUNet models, discovered via TinyNAS, achieve >70% ImageNet top-1 accuracy on devices with <512KB of SRAM.

The Once-For-All and supernet paradigm is a foundational efficiency technique for NAS, especially for scalable deployment.

• Supernet Training: A single, large supernetwork is trained with weight sharing across all possible sub-architectures within the search space.
• Once-For-All Principle: After this one-time training, many efficient subnetworks can be extracted for different hardware constraints (e.g., latency targets of 10ms, 30ms, 100ms) without retraining.
• Benefits for TinyML: Enables a single training run to produce a family of models tailored for various microcontroller tiers (e.g., Cortex-M0, M4, M7), dramatically reducing development cost and time.

A specialized variant of Neural Architecture Search that explicitly optimizes discovered architectures for specific hardware performance metrics. Instead of just accuracy, the search objective incorporates real or proxy measurements of latency, memory usage, energy consumption, or FLOPs on the target device (e.g., a specific microcontroller or NPU). This ensures the final model is not only accurate but also deployable within strict hardware constraints.

A train-once-deploy-everywhere NAS approach where a single, large supernetwork is trained to contain a vast number of smaller, optimal subnetworks within its weight-sharing structure. After one training cycle, specialized models for different latency, memory, or power targets can be extracted instantly without retraining. This is highly efficient for deploying a family of models across a heterogeneous fleet of edge devices.

The overarching discipline of reducing a neural network's computational and storage footprint. It encompasses techniques applied after an architecture is designed (by NAS or humans). Key methods include:

• Quantization: Reducing numerical precision of weights/activations.
• Pruning: Removing redundant parameters.
• Knowledge Distillation: Training a small model to mimic a large one. NAS often discovers architectures that are inherently more amenable to these subsequent compression steps.

A foundational NAS methodology that formulates the search for discrete architectural components (e.g., which operation to use between two layers) as a continuous optimization problem. It uses a supernet with continuous relaxation (softmax over operations) and standard gradient descent. This makes the search significantly more efficient than earlier reinforcement learning or evolutionary-based methods, though it requires substantial memory during the search phase.

A compression technique that removes parameters deemed less important to the model's output. It creates sparse networks with many zero weights. Pruning can be:

• Unstructured: Removes individual weights, requiring specialized hardware/software for speedups.
• Structured: Removes entire channels or filters, leading to directly smaller, hardware-friendly models. NAS can be combined with pruning constraints to search for architectures that are inherently sparse or prune-friendly.

A process where a model is trained or fine-tuned with simulated quantization noise added to the weights and activations. This allows the model to learn parameters that are robust to the precision loss incurred during subsequent deployment to low-bit integer formats (e.g., INT8). QAT is often integrated into the NAS loop (Differentiable Quantization) to discover architectures that perform well under quantized inference from the outset.