Pareto Frontier Search

A candidate neural network architecture is Pareto optimal if no other architecture in the search space is superior in all objectives. Improving one metric (e.g., accuracy) must worsen another (e.g., latency). The set of all Pareto optimal points forms the Pareto frontier, a curve or surface representing the best possible trade-offs. This is distinct from scalarization, which combines objectives into a single weighted sum, as it reveals the full spectrum of optimal solutions.

The search simultaneously optimizes for multiple, often conflicting, objectives. For TinyML and hardware-aware NAS, common objectives include:

• Model Accuracy (e.g., Top-1 % on a validation set)
• Inference Latency (measured in milliseconds on target MCU)
• Peak Memory Usage (RAM and Flash in kilobytes)
• Energy Consumption (microjoules per inference)
• Model Size (parameter count) The algorithm does not return a single "best" model but a diverse set of architectures from different points on the frontier.

Pareto Frontier Search is not a standalone algorithm but a guiding principle integrated into various NAS search strategies:

• Evolutionary Algorithms (NSGA-II, SPEA2): Maintain a population of architectures, using non-dominated sorting and crowding distance to evolve toward the frontier.
• Reinforcement Learning: The controller's reward function is formulated to encourage discovering non-dominated architectures.
• Bayesian Optimization: Models the Pareto frontier probabilistically to suggest promising candidate architectures for evaluation.
• Gradient-Based Methods (e.g., DARTS): Can be extended by optimizing a multi-objective loss that encourages convergence to the frontier.

Accurate, low-cost estimation of hardware metrics is critical. Instead of full on-device profiling for every candidate, the search uses:

• Pre-built Lookup Tables (LUTs): Pre-measured latency/energy costs for each operator (conv3x3, depthwise-separable conv) on the target hardware.
• Analytical Cost Models: Lightweight functions that predict latency/memory based on architecture parameters (e.g., MACs, memory accesses).
• Hardware-in-the-Loop Validation: A subset of promising frontier candidates is profiled on real hardware to ground-truth the estimates and refine the cost model. This allows the search to efficiently navigate billions of potential architectures.

The goal is to approximate the true Pareto frontier with a limited evaluation budget. Key techniques include:

• Non-Dominated Sorting: Ranks architectures based on how many others they dominate, prioritizing those on the current best frontier.
• Diversity Preservation: Mechanisms like crowding distance ensure the discovered frontier points are spread across the objective space, revealing trade-off extremes (e.g., max-accuracy vs. min-latency models).
• Early Stopping & Performance Prediction: Using zero-cost proxies or training for few epochs to cheaply filter out poor candidates before full evaluation.

The final output is a Pareto set—a collection of neural network architectures, each with its associated performance vector. For example:

• Architecture A: 95% accuracy, 50 ms latency, 200 KB memory
• Architecture B: 92% accuracy, 20 ms latency, 100 KB memory
• Architecture C: 88% accuracy, 10 ms latency, 50 KB memory A system designer can then select the specific architecture that best matches their deployment constraints (e.g., "must run in < 15 ms") without sacrificing unnecessary accuracy.

The overarching automated design process that Pareto Frontier Search operates within. It optimizes a neural network architecture across multiple competing objectives simultaneously, such as:

• Model Accuracy
• Inference Latency
• Memory Footprint
• Energy Consumption The goal is not a single 'best' model, but a set of optimal trade-offs, known as the Pareto frontier.

A predictive function critical for efficient Hardware-Aware NAS. Instead of deploying every candidate model on real hardware—which is prohibitively slow—a cost model estimates key metrics:

• Look-Up Tables (LUTs): Pre-measured latencies for standard operations.
• Neural Predictors: Small ML models trained to predict latency or energy from an architecture description. These models allow the search algorithm to approximate the Pareto Frontier without physical profiling for every candidate.

The defined universe of all possible neural network architectures the search algorithm can explore. For TinyML, this space is tightly constrained and includes parameters like:

• Operator Types: Choice between standard convolution, depthwise convolution, or pooling.
• Kernel Sizes: 3x3, 5x5, or other filter dimensions.
• Channel Widths: The number of output filters per layer.
• Skip Connections: Whether to include residual connections. A well-designed search space is fundamental for Pareto Frontier Search to yield viable, hardware-efficient models.

The efficiency engine behind modern NAS. A Supernet is an over-parameterized network that contains all possible pathways defined by the search space.

• Weight Sharing: All candidate sub-architectures share parameters from this single Supernet.
• One-Shot Evaluation: This allows for the approximate performance of thousands of architectures to be assessed without individually training each one from scratch. This technique makes exploring the vast search space for Pareto-optimal points computationally feasible.

The extreme specialization of Hardware-Aware NAS for microcontroller targets. It imposes severe, non-negotiable constraints that shape the Pareto Frontier:

• Memory Budgets: Models must fit within KB of SRAM (runtime) and Flash (storage).
• Energy Limits: Inference may be bounded to microjoule-level consumption.
• Latency Requirements: Real-time operation often demands sub-100ms inference. MCU-NAS searches for architectures where improving accuracy beyond a point is impossible without violating these hard hardware limits.

An advanced paradigm that moves beyond searching for models on fixed hardware. Here, the Pareto Frontier is explored in a joint space:

• Algorithmic Design: The neural network architecture.
• Hardware Design: Features of the accelerator (e.g., memory hierarchy, dataflow). The search optimizes both simultaneously, discovering model-accelerator pairs that achieve superior system-level efficiency than optimizing either component in isolation.