Multi-Objective Neural Architecture Search

The primary output of Multi-Objective NAS is not a single 'best' model, but a set of Pareto-optimal architectures. An architecture is Pareto-optimal if no other architecture is strictly better across all objectives. This creates a trade-off curve (or surface) where improving one metric (e.g., accuracy) necessitates degrading another (e.g., latency). Practitioners select the final model from this frontier based on their specific deployment constraints.

The search explicitly balances inherently competing goals. Common objective pairs include:

• Accuracy vs. Latency: Maximizing predictive performance while minimizing inference time.
• Accuracy vs. Model Size: Maximizing accuracy while minimizing the number of parameters or memory footprint.
• Accuracy vs. Energy Consumption: Critical for battery-powered microcontrollers.
• Latency vs. Peak Memory: Ensuring the model fits within an MCU's SRAM and meets real-time deadlines. The search algorithm must navigate this complex landscape without collapsing to a single-objective optimum.

To optimize for metrics like latency and energy, Multi-Objective NAS relies on hardware cost models. These are predictive functions that estimate the real-world performance of a candidate architecture without deploying it on physical hardware for every evaluation. Methods include:

• Pre-built lookup tables of operator latencies.
• Small neural network predictors trained on profiling data.
• Analytical models based on FLOPs, memory bandwidth, and hardware specs.
• Hardware-in-the-loop profiling for final validation, where select candidates are measured on the actual device.

Standard NAS search strategies are adapted to handle multiple objectives. Key approaches are:

• Scalarization: Combining multiple objectives into a single weighted sum (e.g., reward = accuracy - α * latency). This simplifies the problem but requires careful tuning of weights (α).
• Multi-Objective Evolutionary Algorithms (MOEAs): Using algorithms like NSGA-II to maintain a diverse population of architectures and evolve them towards the Pareto frontier.
• Reinforcement Learning with Multi-Objective Rewards: The controller network receives a reward vector, and strategies like Pareto-based ranking are used for policy updates.
• Gradient-Based Methods with Constrained Optimization: Framing one objective as the primary goal (accuracy) and others as inequality constraints (latency < 10ms).

When targeting microcontrollers (MCU-NAS), the objectives reflect extreme resource scarcity. This involves:

• Kilobyte-scale memory budgets for model weights (flash) and activations (SRAM).
• Microjoule-level energy constraints per inference.
• Real-time latency requirements in the millisecond range. The search space is often pre-pruned to exclude operations incompatible with MCUs (e.g., large dense layers) and may integrate quantization-aware search (QA-NAS) to find architectures robust to 8-bit or lower precision.

A key outcome is a quantitative framework for decision-making. By analyzing the Pareto frontier, engineers can answer critical questions:

• How much accuracy must be sacrificed to reduce latency by 50%?
• What is the minimal memory model that achieves >95% of the baseline accuracy?
• Is there a 'knee' in the curve where large resource gains come with minimal accuracy loss? This data-driven analysis replaces guesswork in selecting the optimal model for a production TinyML deployment, balancing performance with hard physical limits.

The primary predictive performance metric, such as classification accuracy, F1-score, or mean average precision (mAP). In Multi-Objective NAS, accuracy is almost always a primary objective, but it competes directly with efficiency metrics. The search seeks architectures on the Pareto frontier, where any increase in accuracy would cause an unacceptable degradation in another objective like latency or size.

The time required to execute a single forward pass of the model on the target hardware, measured in milliseconds or microseconds. This is a critical real-time constraint for embedded systems.

• Measured via: Direct on-device profiling, a pre-characterized hardware cost model, or cycle-accurate simulators.
• Key Influencers: Model depth, operator types (e.g., standard vs. depthwise convolution), and data movement costs.
• NAS Role: The search algorithm penalizes architectures whose predicted or measured latency exceeds a target budget.

The total memory required to store the model's parameters (weights) and activations. For microcontrollers, this is often the most binding constraint.

• Parameter Memory (Flash): Stored in non-volatile memory. Targets are often < 500 KB for TinyML.
• Activation Memory (RAM): Peak working memory during inference. This is frequently the limiting factor, with budgets as low as < 64 KB.
• NAS Role: The search objective minimizes the number of parameters and the size of intermediate feature maps, often by favoring efficient operations like depthwise convolutions and reducing channel counts.

The total energy used per inference, measured in microjoules (µJ) or millijoules (mJ). This is paramount for battery-powered IoT devices aiming for multi-year operation.

• Directly correlated with the number of arithmetic operations (MACs) and, more importantly, memory accesses. Fetching data from RAM/Flash is often more expensive than the computation itself.
• NAS Role: The search uses an energy cost model to estimate and minimize the energy of candidate architectures. This often leads to designs that minimize data movement and leverage on-chip memory hierarchies effectively.

The number of Multiply-Accumulate operations (MACs) or Floating-Point Operations (FLOPs) required for one inference. While a proxy for latency and energy, it does not account for hardware-specific efficiencies like parallelization or memory bottlenecks.

• Use in NAS: A common, hardware-agnostic objective to initially prune inefficient architectures. It is often combined with a more precise hardware cost model for final candidate selection.
• Limitation: A model with lower MACs is not always faster on real hardware if its operations are not well-supported by the accelerator's kernels.

The stability of a model's accuracy when its parameters and activations are converted from 32-bit floating-point to low-bit integer representations (e.g., int8). This is essential for efficient microcontroller deployment.

• Quantization-Aware NAS (QA-NAS): Integrates simulated quantization (QAT) into the search loop. The objective is to maximize accuracy after quantization.
• Search Outcome: Discovers architectures with redundancy and activation distributions that are inherently robust to precision loss, avoiding sensitive operations that degrade severely when quantized.

A specialized form of NAS that incorporates hardware-specific constraints—such as latency, memory footprint, and energy consumption—directly into the search objective. Unlike standard NAS which optimizes primarily for accuracy, Hardware-Aware NAS discovers models optimized for a target deployment platform (e.g., a specific microcontroller or NPU). It often uses a hardware cost model to predict the on-device performance of candidate architectures without needing to deploy each one.

• Key Objective: Find architectures that meet strict hardware budgets.
• Common Constraints: Inference latency (< 10 ms), peak RAM usage (< 512 KB), model size (< 200 KB).
• Example: Searching for a vision model that achieves >95% accuracy while using less than 100 KB of flash memory on an Arm Cortex-M4.

The optimization goal in Multi-Objective NAS, aiming to discover a set of solutions where no single objective can be improved without degrading another. This results in a Pareto-optimal frontier—a curve or surface plotting the best possible trade-offs (e.g., accuracy vs. latency). The search algorithm's output is not a single "best" model, but a family of models from which a practitioner can select based on their specific resource budget.

• Core Concept: Trade-off analysis between competing metrics.
• Visualization: Often plotted as a 2D/3D frontier of non-dominated points.
• Search Methods: Multi-objective evolutionary algorithms (e.g., NSGA-II), scalarization with different weight vectors.

A predictive function used within Hardware-Aware NAS to estimate the on-device performance of a candidate neural network architecture without full deployment. This model approximates metrics like:

• Latency (inference time)
• Peak Memory (RAM usage)
• Energy Consumption
• Model Size (parameter count)

Cost models can be:

• Analytical: Based on operation counts and hardware lookup tables.
• Learned: Small neural networks trained to predict latency from architecture graphs.
• Hybrid: Combining analytical primitives with learned corrections. Their accuracy is critical for guiding the search toward truly efficient architectures.

An efficient NAS paradigm central to modern Multi-Objective search. A supernet—an over-parameterized network containing all possible operations in the search space—is trained once. Candidate sub-architectures are evaluated by inheriting weights from this shared supernet (weight sharing), avoiding the prohibitive cost of training each candidate from scratch.

• Key Benefit: Drastically reduces search computation from thousands of GPU days to a few.
• Methodologies: Differentiable Architecture Search (DARTS) relaxes the search to be continuous, enabling gradient-based optimization. Once-For-All (OFA) trains a supernet to support diverse sub-networks of varying depths, widths, and kernel sizes.
• Use in Multi-Objective NAS: The supernet enables rapid evaluation of candidates across the accuracy-latency-size trade-off space.

The extreme specialization of Hardware-Aware NAS for microcontroller-class devices. MCU-NAS operates under constraints orders of magnitude tighter than mobile or edge NAS, optimizing for:

• Kilobyte-scale memory (e.g., < 256 KB RAM, < 1 MB Flash)
• Microjoule-level energy per inference
• Single-core, low-clock-speed CPUs (e.g., sub-100 MHz)
• Lack of hardware accelerators (often no GPU/NPU)

Search spaces are designed around microcontroller-friendly operations like depthwise separable convolutions, and cost models must account for severe memory bottlenecks and fixed-point arithmetic overhead. This field is critical for enabling TinyML deployment.

The core algorithm that explores the search space to find high-performing neural architectures. In Multi-Objective NAS, the strategy must balance exploration across multiple competing goals. Primary strategies include:

• Reinforcement Learning (RL-NAS): A controller network generates architectures, receiving a reward based on a weighted sum of multiple objectives (e.g., accuracy - λ * latency).
• Evolutionary Algorithms: A population of architectures evolves via mutation and crossover; selection pressure is based on Pareto dominance or a multi-objective fitness function.
• Gradient-Based (e.g., DARTS): Architecture parameters are continuous and optimized via gradient descent; multiple objectives can be combined in the loss function or used as constraints.
• Bayesian Optimization: Models the performance landscape to suggest promising candidates for expensive evaluation.