TinyMLPerf

TinyMLPerf mandates reporting a balanced set of metrics, avoiding optimization for any single dimension. The core reported metrics are:

• Accuracy: Task-specific metric (e.g., top-1 classification accuracy) measured on the reference dataset.
• Latency: The time to perform a single inference, often reported as tail latency (P90, P99) for real-time systems.
• Throughput: The number of inferences processed per second under a defined load.
• Energy per Inference: Measured in microjoules (µJ), this is the total energy consumed by the hardware to complete one inference.
• Peak Memory Usage: The maximum RAM consumed during inference, critical for constrained MCUs.

To ensure integrity, submissions must adhere to strict submission rules. These rules govern:

• Model fidelity: The model executed must be mathematically equivalent to the reference model within a small numerical tolerance.
• Measurement methodology: Standardized procedures for measuring latency and energy using defined performance counters and external measurement tools.
• Reporting format: A specific JSON schema for results. The compliance checker is an automated tool that validates a submission's logs and results against these rules before they are accepted into the official results repository.

The SUT Interface is the standardized API that the benchmark loadgen uses to communicate with the System Under Test—the combination of hardware, firmware, and software stack being evaluated. This interface:

• Abstracts the specific deployment details (e.g., RTOS, bare-metal).
• Defines functions for loading the model, issuing inferences, and reporting results.
• Allows participants to integrate their highly optimized inference engines (e.g., TensorFlow Lite for Microcontrollers, proprietary kernels) while ensuring they are measured under identical conditions. This enables cross-platform benchmarking of diverse MCU architectures.

Original Equipment Manufacturers (OEMS) and product design firms leverage TinyMLPerf during the hardware selection phase for battery-powered IoT devices. It helps answer critical design questions:

• Which microcontroller meets the accuracy, latency, and power budget for our wake-word detection or anomaly sensing feature?
• What is the real-world battery life impact of continuous inference?
• Does the hardware-software stack deliver deterministic performance required for real-time control?

Developers of TinyML frameworks (e.g., TensorFlow Lite for Microcontrollers, Apache TVM) use TinyMLPerf as a regression test and optimization target. It drives:

• Compiler improvements for kernel libraries and graph optimizations specific to microcontrollers.
• Validation of new quantization schemes and operator implementations.
• Demonstration of the framework's efficiency across a diverse set of benchmark models and hardware backends.

Researchers in academia and industry labs use TinyMLPerf as a common ground for evaluating novel techniques. It provides:

• A reproducible baseline for comparing new model compression algorithms, neural architecture search methods, or on-device learning protocols.
• Metrics to analyze the Pareto frontier between accuracy, latency, and energy for proposed efficient architectures.
• A standardized methodology that ensures research claims are fairly comparable and not skewed by ad-hoc measurement setups.

CTOs, engineering managers, and procurement teams in enterprises deploying large-scale IoT fleets use TinyMLPerf data for vendor-agnostic decision-making. It assists in:

• Creating technical requirements (e.g., "must achieve < 50ms latency on Visual Wake Words") for requests for proposals.
• Auditing vendor performance claims before committing to a hardware platform for thousands of devices.
• Ensuring long-term supply chain flexibility by understanding the performance landscape across multiple silicon vendors.

A benchmark suite is a standardized collection of models, datasets, and measurement procedures designed to evaluate and compare the performance of machine learning systems across different hardware platforms. For TinyML, a suite like TinyMLPerf provides:

• Reference Models: A set of neural networks representing common tasks (e.g., keyword spotting, visual wake words).
• Standardized Workloads: Precisely defined inference tasks with fixed input data.
• Measurement Rules: Strict protocols for timing, power measurement, and accuracy validation to ensure fair comparisons. Its purpose is to move beyond theoretical specs (like TOPS) to measure real-world, end-to-end system performance under controlled, reproducible conditions.

Inference latency is the total time delay, measured from the input of data to the output of a prediction, for a machine learning model to perform a single inference on a target hardware device. In TinyMLPerf, this is a primary latency metric. It is critical for real-time applications (e.g., anomaly detection, voice commands). Measurement must account for:

• Pre/Post-processing: Data formatting and result interpretation.
• Model Execution: The forward pass of the neural network.
• System Overhead: Any RTOS or driver delays. TinyMLPerf reports latency under controlled conditions to isolate the model+hardware performance from external system noise.

Energy per inference is the total electrical energy, typically measured in microjoules (µJ) or millijoules (mJ), consumed by a hardware system to complete a single forward pass of a machine learning model. This is a defining metric for battery-powered TinyML. It integrates power over the entire inference duration. Accurate measurement requires:

• High-precision power meters or integrated current sensors.
• Isolation of the ML workload from background system tasks.
• Stable voltage supply to the microcontroller or accelerator. TinyMLPerf benchmarks this to help developers choose the most energy-efficient hardware-model pairing for a given accuracy target, directly impacting device battery life.

Peak memory usage is the maximum amount of RAM (typically SRAM on an MCU) consumed by a model and its runtime during the execution of an inference task. This includes:

• Model Weights: Stored in Flash but often loaded into RAM for execution.
• Activation Maps: Intermediate layer outputs.
• Runtime Buffers: Space for operators and tensor manipulation. On microcontrollers with only tens to hundreds of kilobytes of SRAM, this metric is a hard constraint. Exceeding available memory causes system failure. TinyMLPerf profiles this to ensure models are feasible for deployment on a given MCU and to guide memory optimization efforts like activation buffering and in-place operations.

The accuracy-latency trade-off describes the fundamental engineering compromise where improving a model's prediction accuracy often increases its computational complexity and inference latency (and energy use), and vice versa. TinyMLPerf benchmarking surfaces this trade-off by testing multiple models or configurations. Engineers use this data to find a Pareto-optimal solution for their application. For example:

• A 99% accurate keyword spotting model may require 500ms of latency.
• A 95% accurate model might run in 50ms. The choice depends on whether the application prioritizes correctness or responsiveness. Benchmarks provide the empirical data to make this decision.

Hardware-in-the-loop testing is a validation methodology where the actual target microcontroller or embedded hardware executes the model and software under test within a simulated or controlled environment. This is the core methodology of TinyMLPerf. Unlike simulation, HIL testing captures real-world effects:

• Actual silicon behavior (cache effects, memory latency).
• True power consumption of the MCU and any peripherals.
• Compiler and toolchain idiosyncrasies. The benchmark runner controls the device, feeds it the golden dataset, and measures results directly from the hardware. This provides the highest-fidelity performance data for deployment decisions.