Benchmark Suite

The core of any benchmark suite is a standardized set of reference models and golden datasets. These provide a consistent workload for cross-platform comparison.

• Models: Typically include a range of common neural network architectures (e.g., MobileNetV1, DS-CNN, TinyConv) covering vision, audio, and anomaly detection tasks.
• Datasets: A golden dataset is a curated, static set of inputs with known expected outputs. It is used for validating both accuracy and performance, ensuring measurements are reproducible and not skewed by data variability.
• Purpose: This standardization allows for an apples-to-apples comparison of how different hardware, compilers, or frameworks handle identical computational graphs and data.

Benchmark suites define a precise set of quantitative metrics to measure system behavior. For TinyML, these extend beyond accuracy to critical resource constraints.

• Core Inference Metrics: Inference latency, throughput (FPS), and peak memory usage (RAM/SRAM).
• Energy Metrics: Energy per inference (in microjoules) is a key differentiator for battery-powered devices.
• Computational Metrics: MACC count (Multiply-Accumulate Operations) measures theoretical workload, while hardware utilization percentages show practical efficiency.
• Advanced Metrics: May include tail latency (P95, P99) for real-time guarantees and worst-case execution time (WCET) analysis for safety-critical systems.

Accurate measurement requires specialized profiling tools that instrument the system with minimal overhead.

• Software Profilers: Tools integrated with frameworks (e.g., TensorFlow Lite Micro profiler) that provide layer-wise profiling of time and memory.
• Hardware Counters: Access to performance counters on the MCU/SoC to measure clock cycles, cache misses, and energy draw from internal sensors.
• External Measurement: Use of precision power monitors (e.g., Joulescopes) and logic analyzers to capture dynamic power and end-to-end latency from sensor to actuation.
• Statistical Profiling may be used on longer-running tests to identify bottlenecks without significant intrusion.

To ensure fairness and reproducibility, benchmark suites enforce strict run rules that define the execution environment and valid results.

• Initialization & Warm-up: Rules for clearing caches and running warm-up inferences to account for one-time costs and achieve steady-state performance.
• Measurement Duration: Specifies the number of inferences to run for a valid measurement, ensuring statistical significance.
• Execution Scenarios: Defines different test conditions, such as:
  • Single-stream latency: Measuring one inference at a time.
  • Multi-task throughput: Measuring sustained inference rate.
  • Power modes: Testing performance in different DVFS (Dynamic Voltage and Frequency Scaling) states.
• Output Validation: Requires that all benchmark runs produce correct outputs against the golden dataset to ensure functional correctness alongside performance.

Comprehensive benchmarking requires full transparency of the system under test (SUT) configuration. This allows for result interpretation and identifies the sources of performance differences.

• Hardware Specification: MCU/NPU model, clock frequency, SRAM/Flash sizes, cache hierarchy, and any dedicated accelerators.
• Software Stack: Operating system (or bare-metal), ML framework version, compiler (and flags used), runtime libraries, and kernel versions.
• Model Implementation Details: The exact model variant (e.g., width multiplier, input resolution), any applied quantization scheme (int8, int16), and the specific optimization passes used (e.g., operator fusion, weight pruning).
• Ambient Conditions: For power measurements, reporting of ambient temperature is often required, as it can affect leakage current (static power) and potential thermal throttling.

A profiling tool is software that instruments a running system to collect detailed data on its runtime behavior. For TinyML, these tools measure metrics critical for optimization, such as cycle-accurate timing, memory access patterns, cache misses, and power draw.

• Function: Provides visibility into bottlenecks at the layer, operator, or even instruction level.
• Examples: ARM Streamline, Segger SystemView, and vendor-specific tools like STM32CubeMonitor.
• Use Case: Essential for identifying whether a model is compute-bound or memory-bound, guiding optimization efforts like kernel fusion or memory layout changes.

Layer-wise profiling is the granular analysis of resource consumption for each individual layer or operator within a neural network. Instead of providing only an overall inference time, it breaks down the contribution of each convolutional layer, fully connected layer, or activation function.

• Key Metrics: Execution time, peak memory usage, and energy consumption per layer.
• Benefit: Pinpoints specific layers that are performance hotspots, allowing for targeted optimizations such as operator substitution, precision reduction, or kernel tuning.
• Output: Often visualized as a timeline or histogram, showing the cost distribution across the network graph.

The roofline model is an analytical performance model that visualizes the attainable performance of a computational kernel (or an entire model) as a function of its operational intensity. It plots performance (e.g., GOP/s) against operational intensity (Ops/Byte).

• Components: Features a compute roof (peak FLOP/s of the hardware) and a memory roof (bandwidth-bound performance).
• Diagnostic Use: A kernel's plotted position reveals its limiting factor:
  • Below memory roof: Performance is limited by memory bandwidth (memory-bound).
  • Below compute roof: Performance is limited by arithmetic units (compute-bound).
• TinyML Application: Guides architects toward optimizing data reuse or selecting more efficient operators.

Hardware-in-the-loop testing is a validation methodology where the actual target microcontroller executes the model and software within a simulated or controlled environment. Sensor inputs are emulated, and outputs are monitored, all while running on the real silicon.

• Purpose: Captures real-world effects like interrupt latency, memory bus contention, and thermal behavior that are absent in pure software simulation.
• Setup: Involves a host PC managing the test bench and communicating with the device under test (DUT) via serial or JTAG.
• Critical for: Verifying deterministic execution, measuring worst-case execution time (WCET), and validating power management states before field deployment.

A golden dataset is a curated, static, and representative set of input data with known, validated expected outputs. It serves as the authoritative reference for evaluating model accuracy and performance during benchmarking and regression testing.

• Characteristics: Must cover the expected input distribution, including edge cases, to ensure robust evaluation.
• Role in Benchmarking: Provides consistent inputs across different hardware platforms and software versions, ensuring fair comparisons.
• Beyond Accuracy: Also used to measure performance invariants like deterministic execution and to profile consistent memory access patterns.