Profiling Tool

TinyML profilers capture a holistic set of performance counters beyond simple timing. Key metrics include:

• Inference Latency: Total time from input to prediction.
• Peak Memory Usage: Maximum RAM/SRAM consumed by weights, activations, and buffers.
• Energy per Inference: Electrical energy consumed, measured in microjoules.
• Hardware Events: Cache misses, instruction retirements, and NPU utilization via on-chip performance counters. This multi-faceted data is essential for identifying bottlenecks, whether a model is compute-bound or memory-bound.

Advanced profilers decompose total resource consumption by individual network components. This layer-wise profiling reveals which specific operations (e.g., a depthwise convolution or fully connected layer) are the primary contributors to latency, memory spikes, or energy draw. This granularity is critical for targeted optimization, such as applying selective quantization or pruning to the most expensive layers to improve the overall model efficiency.

Effective TinyML profilers integrate deeply with the target microcontroller or accelerator's architecture. They leverage:

• Hardware-in-the-Loop (HIL) Testing: Profiling on the actual physical device or accurate cycle-accurate simulators.
• Microarchitectural Events: Tracking events like pipeline stalls and memory bus contention.
• Power Rails Monitoring: Measuring current draw directly from the board's power supply. This ensures profiles reflect real-world behavior, including effects like thermal throttling and deterministic execution variances.

Profilers are not just diagnostic; they are integral to the model development loop. Data feeds directly into optimization processes:

• Roofline Model Analysis: Plotting operational intensity against hardware ceilings to identify bottlenecks.
• Pareto Frontier Exploration: Visualizing the accuracy-latency trade-off or accuracy-energy trade-off to select optimal models.
• Benchmarking: Providing the raw data for standardized suites like TinyMLPerf. This closes the loop between measurement, analysis, and actionable model or code refinement.

For accurate measurement on resource-starved devices, profiling tools must minimize their own footprint. Techniques include:

• Statistical Profiling: Periodically sampling the program counter instead of continuous tracing.
• Selective Instrumentation: Profiling only specific code regions or inference runs.
• Dedicated Debug Hardware: Using on-chip trace modules (e.g., ARM ETM) to export data with minimal CPU involvement. The goal is to measure the system's true performance without significantly altering its behavior, ensuring valid metrics for worst-case execution time (WCET) analysis.

Beyond the model's forward pass, comprehensive profilers capture the entire pipeline. This end-to-end latency measurement includes:

• Sensor data acquisition and preprocessing (e.g., audio MFCC extraction).
• Data movement between memory hierarchies.
• Model inference execution.
• Post-processing and actuation logic. This system-wide view is crucial, as bottlenecks often reside in data handling or system software, not the neural network itself. It also enables analysis of tail latency (P95, P99) for real-time systems.