Hardware Cost Model

The latency predictor estimates the inference time of a candidate neural network on the target hardware. It is the most critical component for real-time applications.

• Mechanism: Often implemented as a lookup table (LUT) pre-populated with layer-wise latency measurements, or as a small neural network (e.g., a multilayer perceptron) trained on profiling data.
• Inputs: Takes architectural parameters (e.g., operator type, input/output channels, kernel size, stride) and hardware state (e.g., CPU/GPU frequency, cache state).
• Output: A single scalar value representing total inference latency in milliseconds or microseconds.
• Example: For a microcontroller, the predictor must account for fixed-point arithmetic overhead and memory access patterns, which dominate execution time.

The energy model predicts the total energy consumption (joules) or average power (watts) required for a single inference. This is paramount for battery-operated edge and TinyML devices.

• Granularity: Can be modeled at the layer level, summing contributions from compute operations (MACs), memory accesses (DRAM, SRAM), and data movement.
• Key Factors: Dynamic power from transistor switching and static leakage power. On microcontrollers, flash memory reads and SRAM accesses are significant contributors.
• Use Case: In Microcontroller NAS (MCU-NAS), the search objective often minimizes microjoules per inference, directly extending device battery life.

This component calculates the peak RAM (working memory) and flash (model storage) requirements of a neural network architecture.

• RAM Estimation: Sums the size of all activation maps, intermediate tensors, and scratch buffers during inference. This determines if the model fits within a microcontroller's limited SRAM (e.g., 256KB).
• Flash Estimation: Calculates the size of the model weights and biases after compression (e.g., quantization). Determines storage requirements on the device.
• Constraint Enforcement: Provides a hard stop in the NAS search loop, immediately pruning architectures that exceed the target hardware's memory budget.

A set of pre-characterized performance data for every primitive operation supported on the target hardware. This is the foundational data for the predictive models.

• Contents: Contains measured latency, energy, and cycle counts for operations like:
  • 3x3 depthwise convolution with 64 input channels
  • 2x2 average pooling
  • Fully-connected layer with 128 inputs, 10 outputs
• Generation: Created via micro-benchmarking—executing and profiling each operation in isolation on the target device (or simulator).
• Integration: Used directly by a lookup-table-based cost model or as training data for a neural network-based predictor.

A module that encodes the dynamic state of the hardware platform into a form the cost model can use. Performance is not static; it depends on concurrent system load.

• Encoded States:
  • CPU/GPU/NPU utilization and frequency scaling
  • Thermal throttling status
  • Memory bus contention
  • For mobile systems, battery level (which affects sustainable peak performance)
• Purpose: Allows the cost model to provide context-aware predictions, estimating performance under realistic, non-idealized conditions.

In Differentiable Architecture Search (DARTS), the discrete hardware cost must be made continuous. A differentiable surrogate model approximates the hardware metric to enable gradient-based optimization.

• Function: Acts as a smooth, continuous approximation of a normally discrete and non-differentiable metric like latency.
• Implementation: Often a small neural network trained to mimic the behavior of the primary lookup-table-based cost model.
• Benefit: Allows architecture parameters (α) to be optimized via gradient descent with respect to both accuracy loss and hardware cost, enabling efficient joint optimization.

These are mathematical functions derived from hardware architecture principles. They predict cost based on operational intensity, memory bandwidth, and compute throughput.

• Key Metrics: Models often calculate FLOPs (Floating Point Operations) and MACs (Multiply-Accumulates) as proxies, but these are poor predictors for memory-bound microcontrollers.
• Roofline Model: A more sophisticated analytical model that plots attainable performance against operational intensity, identifying whether a layer is compute-bound or memory-bound.
• Use Case: Useful for high-level architectural exploration and for hardware where exhaustive profiling is impossible.
• Drawback: Requires deep hardware expertise to formulate and often lacks accuracy for complex, heterogeneous operations.

A machine learning model, typically a small Multi-Layer Perceptron (MLP) or Graph Neural Network (GNN), is trained to predict hardware cost from a numerical encoding of the network architecture.

• Training Data: The predictor is trained on a dataset of (architecture, measured_cost) pairs collected via profiling.
• Advantage: Can capture complex, non-linear interactions between layers and hardware states that LUTs miss.
• Challenges: Requires significant profiling data for training and introduces its own prediction error and computational overhead.
• Research Example: BRP-NAS uses an LSTM-based predictor for latency on mobile CPUs.

In frameworks like DARTS (Differentiable Architecture Search), the hardware cost model must be integrated into the continuous relaxation of the search space to enable gradient-based optimization.

• Method: The latency or energy of each operation choice is incorporated as a penalty term in the loss function. The search optimizes: Loss = CrossEntropy + λ * HardwareCost.
• Challenge: The cost function must be differentiable with respect to the architecture parameters (alphas). This often requires smoothing discrete LUT values.
• Example: FBNet and MobileNetV3 used differentiable search with a latency lookup table to discover efficient models for mobile phones.

The search space defines the universe of all possible neural network architectures an algorithm can explore. It is parameterized by variables such as:

• Layer types (e.g., convolution, depthwise convolution, pooling)
• Connectivity patterns (e.g., skip connections, branching)
• Hyperparameters like kernel size and number of channels. A hardware cost model evaluates candidates sampled from this space. A well-designed search space is crucial; if the optimal architecture isn't within it, the search will fail.

The search strategy is the algorithm that navigates the search space to find high-performing architectures. Common strategies include:

• Reinforcement Learning (RL-NAS): A controller network learns to generate architectures.
• Evolutionary Algorithms: A population of architectures evolves via mutation and crossover.
• Gradient-Based Methods (e.g., DARTS): The search space is relaxed to be continuous and optimized via gradient descent. The hardware cost model is queried repeatedly by the search strategy to evaluate candidates, guiding the search toward hardware-efficient designs.

A performance estimator predicts the final accuracy of a candidate architecture without full training, which is computationally prohibitive in NAS. Techniques include:

• Weight sharing via a supernet.
• Zero-cost proxies that use metrics from the network's initial state.
• Learning curve extrapolation. This is distinct from a hardware cost model, which predicts latency/energy. Together, they form a multi-objective estimator: one for accuracy, one for hardware cost.

Multi-Objective NAS explicitly optimizes for several competing goals simultaneously, such as:

• Model Accuracy
• Inference Latency
• Peak Memory Usage
• Energy Consumption A hardware cost model provides the estimates for the non-accuracy objectives. The search aims to find architectures on the Pareto frontier, where improving one metric worsens another, presenting optimal trade-offs for engineers.

This is a gold-standard methodology where candidate architectures are deployed and profiled on the actual target hardware (e.g., a specific microcontroller) during the search. It provides:

• Ground-truth measurements of latency and energy, avoiding prediction errors from cost models.
• Validation of the cost model's accuracy. Due to its slowness, it's often used to generate a dataset for training a faster, surrogate hardware cost model, which is then used for the bulk of the search.

Neural Hardware Co-Design is a broader, integrated philosophy where the neural network architecture and the hardware accelerator (e.g., a Neural Processing Unit) are jointly optimized. Hardware-Aware NAS is a key technique within co-design. The process may involve:

• Using NAS to find optimal architectures for a given hardware template.
• Modifying hardware features (e.g., memory hierarchy, dataflow) based on common patterns discovered by NAS. This achieves peak system-level efficiency beyond optimizing either component in isolation.