MLPerf Tiny

MLPerf Tiny is explicitly designed for microcontroller-class devices (MCUs) and other deeply embedded systems. The benchmark assumes severe constraints typical of the TinyML domain:

• Memory: Often less than 512KB of SRAM and a few MB of Flash.
• Compute: Single or dual-core Arm Cortex-M class CPUs, often without a floating-point unit (FPU).
• Power: Milliwatt-scale operational budgets, targeting always-on, battery-powered applications. This focus distinguishes it from other MLPerf benchmarks for mobile phones, datacenter accelerators, or edge servers.

The suite comprises four core, production-relevant tasks that represent common on-device inference workloads:

• Keyword Spotting: Detecting spoken commands from the Google Speech Commands dataset.
• Visual Wake Words: Determining if a person is present in an image from the Visual Wake Words dataset.
• Image Classification: Classifying images from the CIFAR-10 dataset.
• Anomaly Detection: Identifying irregularities in machine sensor data from the ToyADMOS dataset. These tasks cover audio, vision, and sensor modalities, providing a holistic view of a system's capabilities.

MLPerf Tiny measures systems across three primary, hardware-agnostic metrics to evaluate the trade-offs inherent in edge deployment:

• Accuracy: The primary quality metric (e.g., classification accuracy, F1 score). Submissions must meet a minimum accuracy threshold.
• Latency: The inference latency for a single sample, measured in milliseconds. This determines real-time responsiveness.
• Energy: The total energy consumed per inference, measured in microjoules (µJ). This is critical for battery life. Results are presented in a multi-dimensional score, preventing optimization for any single metric at the expense of others.

To ensure fair comparisons, the benchmark defines strict submission categories that dictate what can be optimized:

• Closed Division: Models must use the reference model architecture provided for each task. Only post-training techniques like model quantization, pruning, and compiler optimizations are allowed. This tests hardware and software stack efficiency.
• Open Division: Participants can submit any model architecture, including those found via Hardware-Aware Neural Architecture Search (NAS), provided it fits within the memory footprint limit. This fosters innovation in efficient model design. All submissions must run entirely on the target device—no split inference to external servers is permitted.

MLPerf Tiny serves as a critical inference performance benchmarking tool for multiple stakeholders:

• Hardware Vendors: Use it to demonstrate the ML performance-per-watt of new MCUs, Neural Processing Units (NPUs), and DSPs.
• Software Framework Developers: Optimize runtimes like TensorFlow Lite for Microcontrollers and Apache TVM to generate better code for the benchmark tasks.
• System Integrators: Provides objective data to select the optimal hardware/software stack for a product.
• Researchers: Offers a standardized, reproducible testbed for evaluating new model compression and optimization techniques.

MLPerf Tiny is a specialized track within the larger MLPerf ecosystem, with key differentiators:

• Scale: Focuses on sub-1 watt devices, whereas other tracks target watts to kilowatts.
• Workloads: Uses small, fixed tasks; other tracks use massive datasets like ImageNet or MLPerf Inference uses full-scale models like BERT.
• Metrics: Uniquely includes energy consumption as a first-class metric.
• Deployment Philosophy: Embodies the TinyML principle of extreme efficiency, versus the high-throughput, batch-oriented focus of datacenter benchmarks. It provides the missing standardized evaluation for the rapidly growing edge AI and microcontroller market.

MLPerf Tiny evaluates submissions on a multi-dimensional scorecard, emphasizing the trade-offs inherent in edge deployment.

• Accuracy: Primary metric (e.g., Top-1 % for classification tasks). Must meet a minimum threshold to be valid.
• Latency: Measured in milliseconds per inference on the target hardware. Critical for real-time responsiveness.
• Energy: Measured in microjoules per inference. This is the defining constraint for battery-powered devices.
• Memory Footprint: Model size in kilobytes (RAM for runtime, Flash for storage). Determines the class of MCU required.
• Result Reporting: Submissions are categorized into closed division (strict model and dataset rules) and open division (for research innovation).

The benchmark tests the full system, not just the model, requiring co-optimization across the entire stack.

• Target Hardware: Microcontrollers (e.g., Arm Cortex-M series, ESP32, Arduino Nano 33 BLE Sense), often with clock speeds <200 MHz and RAM <512KB.
• Software Frameworks: TensorFlow Lite for Microcontrollers, Apache TVM, CMSIS-NN (Arm's optimized neural network kernels).
• Compilation & Deployment: The process involves quantization (often to INT8), pruning, and using hardware-specific compilers to generate efficient C/C++ code that runs without an OS.
• System Integration: The benchmark measures end-to-end performance, including sensor data ingestion and pre-processing on the device.

MCU deployment is the target environment for MLPerf Tiny. Microcontrollers are single-chip computers with integrated memory, processor, and I/O. Key constraints include:

• SRAM: Typically 32KB to 512KB for storing model weights and activations during inference.
• Flash: 128KB to 2MB for storing the model binary and application code.
• Lack of OS: Models often run on bare-metal or real-time operating systems (RTOS), requiring highly optimized, static memory allocation.

Deploying to MCUs requires specialized toolchains like TensorFlow Lite for Microcontrollers and CMSIS-NN libraries.

Model quantization is a critical compression technique for MLPerf Tiny submissions. It reduces the numerical precision of model weights and activations from 32-bit floating-point (FP32) to lower-bit formats like INT8 or INT4. Benefits are essential for MCUs:

• Memory Reduction: INT8 weights use 75% less memory than FP32.
• Compute Speedup: Integer arithmetic is faster and more energy-efficient on hardware without FPUs.
• Maintained Accuracy: Techniques like Quantization-Aware Training (QAT) or post-training quantization minimize accuracy loss.

Most high-scoring MLPerf Tiny models use INT8 quantization.

A Neural Processing Unit is a specialized hardware accelerator designed to execute the matrix and tensor operations fundamental to neural networks with extreme energy efficiency. For the edge:

• Integrated NPUs: Found in advanced microcontrollers (e.g., Arm Ethos-U55/U65) and application processors, offering orders of magnitude better performance-per-watt than a CPU.
• MLPerf Tiny Role: Submissions often use NPUs to achieve the highest scores. The benchmark measures the full system performance, including hardware acceleration.
• Compiler Support: Frameworks like Apache TVM and TensorFlow Lite Micro must compile models to specialized NPU instruction sets.

Keyword Spotting is one of the four benchmark tasks in the MLPerf Tiny suite (specifically, the Google Speech Commands dataset). It involves identifying a small set of spoken words from a continuous audio stream.

• Model Architecture: Typically uses a Depthwise Separable Convolutional Neural Network to minimize parameters.
• Real-time Requirement: Must process audio frames with very low latency to enable responsive voice interfaces.
• Use Case: Found in smart home devices, wearables, and always-on assistants. It is a quintessential TinyML task due to its need for low-power, continuous operation.

Visual Wake Words is another core MLPerf Tiny benchmark task. The goal is to classify whether a person is present in a low-resolution (96x96) image from a camera.

• Efficiency Challenge: Requires compressing a vision model like MobileNetV1 to under 250KB while maintaining >90% accuracy.
• Use Case: Enables privacy-preserving, always-on sensing for security cameras, smart displays, and IoT devices, triggering higher-power systems only when needed.
• Architecture Search: This task drives innovation in hardware-aware neural architecture search (NAS) to find optimal model architectures for a given MCU+NPU combination.