Performance Estimator (NAS)

Zero-cost proxies are metrics that estimate a neural network's trainability or final performance using only its initialization state, requiring no training and minimal computation. They analyze properties like gradient flow or synaptic strength at initialization.

• Examples: NASWOT (Network Architecture Search Without Training), grad_norm, synflow.
• Use Case: Ultra-fast, initial screening of massive search spaces to prune poor candidates before applying more expensive estimators.
• Limitation: Lower correlation with final accuracy compared to methods that use some training.

This method trains a single, over-parameterized supernet containing all possible operations in the search space. Candidate sub-architectures are evaluated by sharing the supernet's weights, avoiding independent training.

• Mechanism: The supernet is trained with techniques like path dropout. A candidate's performance is estimated by activating its specific path and evaluating on a validation set.
• Advantage: Enables evaluation of thousands of architectures at the cost of training one network.
• Key Challenge: Optimization bias where the supernet training may not perfectly reflect stand-alone training of sub-networks.

A hardware cost model is a predictive function that estimates latency, energy, or memory usage of a candidate architecture on specific target hardware. It is essential for Hardware-Aware NAS.

• Types: Lookup tables (LUTs) pre-measuring operator costs, or small neural network regressors trained on profiling data.
• Integration: The cost prediction is combined with accuracy estimation in a multi-objective search (e.g., Accuracy / (Latency)^λ).
• Fidelity: Highest accuracy comes from hardware-in-the-loop profiling, but this is slower than purely analytical models.

This approach estimates final performance by training candidates on a subset of data, for fewer epochs, on a lower resolution, or on a simpler proxy task.

• Rationale: Performance rankings on cheap proxies often correlate with rankings from full training.
• Example: Training ImageNet candidates on CIFAR-10 first, or for only 5 epochs instead of 200.
• Trade-off: Provides a better signal than zero-cost proxies but is more computationally expensive. Critical for balancing search speed and estimation quality.

A predictor-based method uses a separate model (e.g., a hypernetwork or regressor) to map a candidate architecture's encoding directly to a predicted performance score.

• Process: 1) Sample and fully train a set of architectures. 2) Train the predictor on {architecture encoding, final accuracy} pairs. 3) Use the trained predictor to score new, unseen candidates.
• Strength: Can be very fast at query time after the initial training investment.
• Consideration: Prediction quality depends heavily on the diversity and quality of the training data for the predictor.

The most direct and accurate method, Hardware-in-the-Loop Search deploys, compiles, and profiles candidate architectures on the actual target hardware (e.g., a specific microcontroller) during the search.

• Output: Provides ground-truth measurements for latency, peak memory usage, and energy consumption.
• Advantage: Eliminates error from analytical hardware models; critical for extreme constraints where cache effects and memory alignment dominate.
• Cost: Significantly slower than purely software-based estimation, often used to validate or fine-tune results from faster estimators.

Zero-cost proxies are lightweight metrics that estimate a network's trainability or final performance using only its initial, untrained state. They require zero training epochs and minimal computation, making them ideal for the initial screening of thousands of candidates in a TinyML search.

• Examples: Metrics like gradient norm, synaptic flow (SNIP), and Fisher Information.
• Use Case: Pruning a massive search space of potential microcontroller-compatible architectures down to a few hundred promising candidates before applying more expensive estimators.

A hardware cost model is a predictive function that estimates key deployment metrics—latency, energy consumption, and memory footprint—for a given architecture on target silicon. For microcontrollers, these models are essential for ensuring a discovered model fits within kilobyte-scale memory budgets.

• Lookup Tables (LUTs): Pre-measured costs for individual operations (e.g., 3x3 depthwise convolution) are summed to estimate total cost.
• Neural Predictors: Small MLPs trained on architecture-performance pairs to predict costs for novel, unseen layer combinations.
• Accuracy: Ground-truth measurement via hardware-in-the-loop profiling is used to build and validate these models.

This approach trains a single, over-parameterized supernet that encompasses all possible architectures in the search space. Candidate performance is estimated by inheriting weights from the supernet, eliminating the need to train each candidate from scratch.

• Mechanism: The supernet is trained with techniques like path dropout or sandwich rule sampling.
• Efficiency: Enables evaluating thousands of sub-networks in seconds by simply activating different paths through the shared weight matrix.
• Framework Example: Once-For-All (OFA) trains one supernet to supply many hardware-tailored sub-networks, a cornerstone for scalable TinyML NAS.

In TinyML, models are almost always quantized (e.g., to 8-bit integers). A quantization-aware estimator predicts how a candidate architecture will perform after this compression, which can drastically affect accuracy.

• Method: The estimator simulates quantization during the search, often using Quantization-Aware Training (QAT) within the supernet.
• Objective: Discovers architectures that are naturally robust to the precision loss from quantization, avoiding post-search accuracy collapse.
• Result: Yields models that maintain high accuracy when deployed to microcontrollers using fixed-point arithmetic libraries like TensorFlow Lite for Microcontrollers.

TinyML NAS must balance competing goals: accuracy vs. latency vs. memory vs. energy. Multi-objective search uses performance estimators for each metric to find the Pareto frontier—a set of architectures where no metric can be improved without worsening another.

• Trade-off Visualization: Results in a curve of optimal models, allowing engineers to select the best architecture for their specific MCU's constraints.
• Search Algorithms: Often uses evolutionary algorithms or Bayesian optimization to navigate this multi-dimensional space efficiently.
• Outcome: Provides a portfolio of viable models for different product configurations (e.g., battery-powered vs. mains-powered devices).

Performance estimators for MCU-NAS must model extreme, atypical constraints not found in mobile or cloud NAS.

• Peak Memory Footprint: Precisely estimates activation memory (RAM) and model weights (Flash) to avoid out-of-memory errors on devices with < 512KB RAM.
• Energy-Per-Inference: Models total energy draw, critical for battery-operated sensors that must run for years.
• Support for Specialized Operations: Accounts for the efficiency of depthwise convolutions and fully connected layers on ARM Cortex-M cores versus standard convolutions, which are prohibitively expensive.

The search space defines the universe of all possible neural network architectures a NAS algorithm can explore. It is parameterized by variables such as:

• Layer types (e.g., convolution, pooling, attention)
• Connectivity patterns (e.g., skip connections, branching)
• Hyperparameters (e.g., number of filters, kernel size)

A well-designed search space balances expressiveness (to find good models) with tractability (to make search feasible). Performance estimators are trained to predict outcomes within a defined search space.

The search strategy is the algorithm that navigates the search space to find high-performing architectures. Common strategies include:

• Reinforcement Learning (RL-NAS): A controller network proposes architectures rewarded for high performance.
• Evolutionary Algorithms: A population of architectures evolves via mutation and crossover.
• Gradient-Based Methods (e.g., DARTS): Architecture parameters are relaxed and optimized via gradient descent.

The search strategy uses the performance estimator to evaluate candidates without full training, guiding the search efficiently.

A supernet is an over-parameterized neural network that embeds all possible operations and pathways defined by the search space. Weight sharing is the technique where all candidate sub-architectures share the parameters of this single supernet.

This is foundational for One-Shot NAS methods. The performance estimator's role is often to rank sub-architectures by evaluating their performance using the shared weights, avoiding the cost of training each from scratch.

A hardware cost model is a predictive function that estimates the on-device resource consumption of a candidate architecture, such as:

• Latency (inference time)
• Peak Memory Usage (RAM/Flash)
• Energy Consumption (joules per inference)

These models can be analytical (e.g., lookup tables based on operator counts) or learned (small neural networks). In Hardware-Aware NAS, the performance estimator often combines predicted accuracy with the output of the hardware cost model to evaluate the Pareto-optimality of a candidate.

Zero-cost proxies are a class of ultra-efficient performance estimators that predict network trainability or final accuracy using only the initial, untrained state of the model. They require no gradient steps and minimal computation.

Examples include:

• grad_norm: The norm of gradients after one mini-batch.
• snip: Sensitivity of the loss to pruning each parameter.
• synflow: A measure of weight synergy.

These proxies are used for rapid, preliminary architecture screening before applying more accurate but costly estimators.

Multi-Objective NAS optimizes for several competing goals simultaneously, such as accuracy, latency, and model size. The output is not a single best model, but a set of Pareto-optimal architectures.

An architecture is Pareto-optimal if no other architecture is better in all objectives. The performance estimator in this context must provide a vector of predictions (e.g., accuracy and latency) for each candidate. The search algorithm then uses this to navigate and populate the Pareto frontier, the curve representing the best possible trade-offs.