TinyML Benchmark Suite

A standardized format for submitting benchmark results, ensuring they are complete, comparable, and verifiable. This defines the "rules of the game" for the benchmark.

• Required fields: All performance metrics (accuracy, latency, memory, energy), exact hardware specifications (MCU, clock speed), software stack versions (compiler, framework), and measurement methodology.
• Reproducibility mandates: Submission of the exact model file (.tflite, .onnx) and the scripts used to run the benchmark.
• Public results dashboard that ranks submissions by different metrics (e.g., accuracy per energy unit), highlighting Pareto-optimal solutions.

This open-source benchmark, often associated with research from ETH Zurich, provides a comprehensive profiling tool for microcontrollers. It goes beyond inference to include training metrics.

• Profiling Depth: Measures memory footprint (RAM/Flash), inference time, and energy use.
• Training Evaluation: Can benchmark the cost of on-device training steps, crucial for federated edge learning research.
• Modular Design: Users can integrate custom models and datasets into the benchmarking pipeline.
• Output: Generates detailed reports for model-hardware co-design optimization.

Specialized benchmarks are emerging to evaluate the unique challenges of Federated Learning (FL) on TinyML devices, moving beyond single-device inference.

• Key Metrics: Measures communication cost (bytes per round), convergence time over federated rounds, and energy per global update.
• System Heterogeneity: Evaluates performance under realistic conditions like partial participation, straggler devices, and non-IID data distributions.
• Frameworks: Tools like Flower and FedML provide built-in benchmarking capabilities for simulating federated scenarios across heterogeneous clients.
• Goal: Quantifies the trade-offs between learning accuracy, privacy, and total system resource consumption.

Benchmark suites focus on neural network architectures and tasks that are commercially relevant for embedded deployment.

• Common Model Architectures:
  • MobileNetV1/V2: For efficient image classification.
  • Deep Neural Network (DNN) & Convolutional Neural Network (CNN) variants for keyword spotting.
  • Autoencoders: For anomaly detection on sensor data.
• Representative Tasks:
  • Visual Wake Words: e.g., "Is there a person in the frame?"
  • Keyword Spotting: e.g., detecting "yes," "no," "up," "down."
  • Anomaly Detection: Identifying irregularities in machine vibration or audio signals.
• Purpose: Ensures benchmarks reflect real-world TinyML applications in IoT, wearables, and industrial monitoring.

Tiny Machine Learning (TinyML) is the overarching field focused on developing and deploying machine learning models on microcontroller units (MCUs) and other deeply embedded devices. It is characterized by extreme constraints:

• Memory Footprint: Models must often fit in < 1 MB of RAM/Flash.
• Energy Budget: Operations must consume microwatts to milliwatts to enable year-long battery life.
• Compute Constraint: Processors typically run at < 200 MHz with no floating-point unit. A benchmark suite provides the standardized tests to evaluate solutions within this specific paradigm.

On-device training is the process of updating a model's parameters directly on an edge device using local data. This is a key capability for federated edge learning, enabling personalization and continual learning without raw data egress. Benchmarking this process is critical and measures:

• Convergence speed given limited local epochs.
• Memory overhead for storing optimizers and intermediate gradients.
• Energy consumption per training step.
• Accuracy recovery after learning from new, on-device data streams.

Model compression techniques are essential to make models viable for TinyML. A benchmark suite evaluates their effectiveness and trade-offs on target hardware. Key techniques include:

• Quantization: Reducing numerical precision of weights/activations (e.g., FP32 to INT8). Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) are standard benchmark components.
• Pruning & Sparsification: Removing less important weights or neurons to create sparse models.
• Knowledge Distillation: Training a small 'student' model to mimic a larger 'teacher'. Benchmarks measure the resulting accuracy, latency, and memory footprint.

An embedded Federated Learning runtime is the lightweight software library on a microcontroller that executes the client-side protocol. It manages:

• Downloading the global model.
• Performing local training with the on-device dataset.
• Applying secure aggregation or differential privacy protocols.
• Uploading model updates. Benchmarking a runtime involves measuring its memory overhead, reliability across availability windows, and efficiency in handling sparse updates to reduce communication costs.

Heterogeneous clients are a core challenge for federated edge learning and its benchmarks. Devices vary in:

• MCU architecture (Arm Cortex-M series, RISC-V).
• Clock speed and presence of neural processing unit accelerators.
• Memory (SRAM, Flash) size and hierarchy.
• Power envelope and connectivity. A robust benchmark suite must run across a matrix of these hardware platforms to report performance portability and identify the straggler problem where slowest devices limit system efficiency.

The TinyML stack is the full software/hardware architecture for deploying ML on microcontrollers. Benchmarks validate each layer's performance:

1. Model Design: Frameworks like TensorFlow Lite for Microcontrollers.
2. Hardware-Aware Compilers: e.g., Apache TVM, STM32 Cube.AI, which perform operator fusion and memory planning.
3. Microcontroller Runtimes: Low-level kernels for optimized integer-only inference.
4. Embedded OS: e.g., FreeRTOS, Zephyr, managing tasks and resources. Benchmarks ensure the entire stack works cohesively under real compute constraints and energy budgets.