Throughput (Frames per Second)

Throughput measures the sustained processing rate over time, not a single burst. It accounts for the entire inference pipeline, including data movement, preprocessing, and postprocessing. For microcontroller systems, this is often measured over thousands of inferences to ensure stability.

• Peak FPS is the theoretical maximum under ideal, isolated conditions.
• Sustained FPS is the practical, achievable rate in a real application loop, which is lower due to system overhead and resource contention.
• A system bottlenecked by memory bandwidth may show a large gap between peak and sustained throughput.

Throughput and inference latency are intrinsically linked but measure different things. For a batch size of 1, throughput is approximately the inverse of latency (FPS ≈ 1 / latency in seconds).

• High latency directly caps maximum throughput.
• Techniques like pipelining or continuous batching can improve throughput without reducing single-inference latency by overlapping computation and data transfer.
• In real-time systems, a target throughput requirement directly defines the maximum permissible latency for each inference.

Processing inputs in batches is a primary method for increasing throughput on capable hardware by amortizing fixed overheads across multiple samples.

• Larger batches improve hardware utilization (e.g., of NPU vector units) and reduce per-inference cost.
• However, batching increases peak memory usage (for storing all batched activations) and tail latency (as the first result must wait for the entire batch).
• On extreme edge devices, batch size is often 1 due to severe SRAM limitations, making throughput optimization reliant on kernel and memory layout efficiency.

Throughput scales with available hardware resources until it hits a saturation point determined by the system's bottleneck.

• Compute-Bound Systems: Throughput increases with higher clock speed or more parallel units until the memory subsystem cannot feed data fast enough.
• Memory-Bound Systems: Throughput is limited by the bandwidth to external flash (for weights) or SRAM (for activations). Optimization focuses on data reuse and cache locality.
• The Roofline Model is used to visualize this relationship between operational intensity and attainable throughput.

For battery-operated devices, throughput is not an independent metric but is governed by a power budget.

• Higher throughput typically requires higher dynamic power consumption.
• Thermal throttling can cause throughput to drop over time as the device heats up.
• The key efficiency metric is inferences per joule, which combines throughput and energy use. An optimized system maximizes throughput within a strict power envelope, often by lowering voltage and clock frequency.

Accurate throughput measurement requires careful methodology to be meaningful for comparison.

• Use a golden dataset representative of real inputs.
• Measure over a long duration (e.g., 10,000 inferences) to account for cache warm-up and OS scheduling jitter.
• Report both average FPS and variance (or tail latency P99).
• Standardized benchmarks like TinyMLPerf provide controlled procedures to measure sustained throughput across different platforms, ensuring fair comparison.

Inference latency is the total time delay from input presentation to prediction output for a single inference. While throughput measures aggregate rate, latency measures individual response time. In real-time systems, low latency is often more critical than high throughput. Key factors include:

• Model architecture complexity
• Hardware clock speed and pipeline efficiency
• Memory access patterns and cache utilization
• Deterministic execution is required for hard real-time guarantees.

Peak memory usage is the maximum RAM/SRAM consumed during inference, encompassing model weights, activations, and intermediate buffers. On microcontrollers with kilobytes of memory, this is a primary constraint. Memory management techniques include:

• Layer-wise profiling to identify memory bottlenecks
• Activation buffer reuse and in-place operations
• Quantization to reduce weight precision (e.g., INT8 vs. FP32)
• External memory interfaces for larger models

Energy per inference measures total electrical energy (µJ/mJ) consumed per forward pass. For battery-powered edge devices, this directly determines operational lifetime. Energy consumption comprises:

• Dynamic power from transistor switching during computation
• Static power from leakage current when idle
• Energy of data movement (often dominates compute energy)
• Optimization involves minimizing MACC count, efficient dataflow, and leveraging low-power sleep states between inferences.

Model efficiency is a holistic metric balancing accuracy against resource consumption (latency, memory, energy). No single metric tells the full story. Evaluation involves:

• Plotting the accuracy-latency trade-off curve
• Identifying the Pareto frontier of optimal designs
• Using composite scores like accuracy/(latency×energy)
• Frameworks like TinyMLPerf provide standardized efficiency benchmarks across hardware platforms.

A system is compute-bound when performance is limited by arithmetic unit speed (e.g., CPU/ NPU peak FLOPS). It is memory-bound when limited by data movement bandwidth. Analysis uses the Roofline Model:

• Operational intensity (Ops/Byte) determines the limiting factor
• Low operational intensity kernels (e.g., fully connected layers) are typically memory-bound
• High operational intensity kernels (e.g., large convolutions) can be compute-bound
• Optimization strategies differ fundamentally for each regime.

End-to-end latency encompasses the total time from sensor trigger to final actuation, including:

• Sensor sampling and signal preprocessing
• Data formatting and buffering
• Model inference time (the core inference latency)
• Post-processing and decision logic
• Output/actuation signaling For true system performance, this holistic measure is more relevant than isolated inference throughput, especially in closed-loop control applications.