MLPerf Tiny

MLPerf Tiny is explicitly designed for microcontroller units (MCUs) and other deeply embedded processors, typically characterized by:

• Severe memory constraints (often < 1 MB of SRAM/Flash)
• Extremely low power budgets (milliwatt-scale operation)
• Limited compute (single-core Arm Cortex-M class CPUs, often without an FPU)
• Lack of an OS or running a minimal RTOS This distinguishes it from other MLPerf benchmarks (like Mobile or Datacenter) which target smartphones, laptops, or servers with orders of magnitude more resources.

The suite comprises a small set of representative TinyML tasks chosen for their real-world relevance and diversity of computational patterns. The current v1.1 benchmarks are:

• Keyword Spotting (KWS): Identify spoken commands from audio.
• Visual Wake Words (VWW): Detect the presence of a person in an image.
• Image Classification (IC): Classify images from the CIFAR-10 dataset.
• Anomaly Detection (AD): Identify anomalous machine sounds from audio. Each task provides a standardized dataset, a reference model, and a precise accuracy target that must be met for a valid submission, ensuring fair comparison.

Performance is measured across several critical axes for embedded systems, not just raw speed:

• Latency: The time to perform a single inference, measured in milliseconds.
• Throughput: The number of inferences processed per second.
• Energy: The total joules consumed per inference, a key metric for battery-powered devices.
• Peak Memory Usage: The maximum SRAM (temporary) and Flash (persistent) memory consumed by the model and runtime.
• Accuracy: The model's task performance, which must meet the benchmark's minimum threshold. Results are presented in a results table that allows engineers to trade off these dimensions based on their application's needs (e.g., lowest energy vs. highest accuracy).

To ensure credibility and prevent unfair optimization, MLPerf Tiny enforces rigorous submission rules:

• Closed Division: Submissions must use the benchmark's official datasets and models; no architectural changes or extra training data are allowed. This tests deployment efficiency.
• Open Division: Allows model architecture changes and retraining, fostering innovation in model design for constrained hardware.
• Required Measurements: Latency and energy must be measured on physical hardware, not simulated.
• Auditability: All submissions include detailed configuration files, code, and measurement methodologies that are reviewed by the MLPerf organization.

The benchmark is platform-agnostic, enabling fair competition across diverse hardware and software stacks:

• Hardware: Supports any MCU, SoC, or accelerator (e.g., Arm Cortex-M, RISC-V, Ethos-U55 NPU).
• Frameworks: Compatible with any TinyML inference engine (e.g., TensorFlow Lite Micro, CMSIS-NN, proprietary vendor SDKs).
• Reference Implementations: Provides a baseline implementation using TensorFlow Lite for Microcontrollers to lower the entry barrier. This agnosticism drives innovation across the entire TinyML ecosystem, from silicon vendors to compiler developers.

Beyond mere measurement, MLPerf Tiny serves as a catalyst and reference point for the TinyML industry:

• Vendor Benchmarking: Chipmakers (ST, NXP, Renesas, etc.) and IP providers (Arm) use it to showcase hardware capabilities.
• Toolchain Validation: Framework developers (TF Lite Micro, TVM) use it to verify optimization passes and compiler correctness.
• Research Benchmark: Academics and researchers use it as a standard testbed for new model compression, neural architecture search (NAS), and efficient kernel techniques.
• Purchasing Guidance: Provides CTOs and engineers with objective, audited data for hardware and software selection.

MLPerf Tiny measures systems across multiple, equally important axes to provide a holistic view of TinyML performance.

• Accuracy: Primary metric (e.g., top-1 accuracy for IC, F1 score for AD). The benchmark defines minimum accuracy targets.
• Latency: Time to perform a single inference, critical for real-time responsiveness.
• Energy: Total joules consumed per inference, measured directly on the hardware under test.
• Memory Footprint: Model size (ROM) and peak RAM usage for activations. These hard constraints define what is deployable.

A system co-design framework that jointly optimizes the neural network architecture (TinyNAS) and the inference runtime (TinyEngine) to enable ImageNet-scale classification on microcontrollers with under 512KB of memory. It represents the state-of-the-art in hardware-aware model and runtime co-optimization.

• TinyNAS searches for models that fit within a device's SRAM/Flash budget.
• TinyEngine generates specialized, ultra-lean C code via operator fusion and in-place depthwise convolution to minimize memory overhead.

A dedicated hardware accelerator, such as the Arm Ethos-U55 or U65, integrated into a microcontroller or system-on-chip to offload and dramatically accelerate neural network inference tasks. MLPerf Tiny benchmarks performance on systems using these accelerators.

• Requires a vendor-specific NPU SDK for model compilation and deployment.
• Executes quantized (int8) models with extreme power efficiency.
• Works alongside a main Cortex-M CPU, which handles control logic and pre/post-processing.

The end-to-end process of converting a trained model into firmware running on a microcontroller, which is the ultimate goal measured by MLPerf Tiny. This involves a specialized TinyML toolchain.

• Model Conversion & Optimization: Using a micro-compiler (e.g., TVM, nncase) for graph optimization and quantization.
• Code Generation: Outputting a C array model or linked library.
• Integration & Profiling: Embedding the model into firmware, managing the tensor arena, and validating latency/accuracy on real hardware.