Utilization (Compute, Memory)

Compute utilization measures the percentage of time a processor's arithmetic logic units (ALUs), DSP cores, or neural processing unit (NPU) macrocells are actively engaged during inference. On microcontrollers, this is rarely a simple average.

• Bursty vs. Sustained: Inference often occurs in short, intense bursts followed by idle sleep states, making average utilization misleading for battery life calculations.
• Parallelism Limits: Utilization of multi-core MCUs is complicated by model parallelism and data dependencies between layers.
• Hardware Acceleration: Utilization of dedicated NPUs or DSPs is a key metric, as their peak operations per second (OPS) defines the system's maximum attainable throughput. Low utilization here often indicates inefficient data movement or kernel scheduling.

This metric tracks the percentage of the available memory bus bandwidth being used to move model weights, activations, and intermediate buffers. In TinyML, systems are often memory-bound, not compute-bound.

• The Memory Wall: Accessing SRAM or Flash consumes orders of magnitude more energy than a multiply-accumulate (MAC) operation. High bandwidth utilization can dominate the energy per inference.
• Data Reuse & Locality: Techniques like layer fusion and tiling aim to maximize data reuse in caches or scratchpad memory, reducing off-chip bandwidth utilization and improving efficiency.
• Measuring Bottlenecks: Profilers use performance counters to correlate high bandwidth utilization with pipeline stalls, identifying layers where compute units are idle waiting for data.

This refers to the percentage of the microcontroller's limited SRAM (often 256KB-1MB) occupied by the model's runtime artifacts. It is a hard constraint for deployment.

• Activation Memory: The memory required to store intermediate layer outputs (activations) often exceeds the size of the static model weights. This is the primary driver of peak SRAM usage.
• Static vs. Dynamic Allocation: Weights are typically stored in Flash and streamed into SRAM or a cache. Activation buffers are dynamically allocated in SRAM during the inference graph execution. Profiling must capture the peak memory usage snapshot.
• Memory Pooling: Advanced runtimes use static memory planning to allocate a single, reusable block for all activations, minimizing fragmentation and ensuring deterministic execution.

Utilization directly drives dynamic power consumption, following the formula P_dynamic = α * C * V^2 * f, where activity factor (α) correlates with utilization.

• Linear Scaling: Higher compute and memory bandwidth utilization typically leads to linearly higher dynamic power draw.
• Static Power Overhead: Even at 0% compute utilization, the powered-on MCU consumes static power due to leakage. This makes low-duty-cycle operation critical for battery life.
• Thermal Implications: Sustained high utilization on a small die can lead to thermal throttling, where the clock speed is reduced to prevent overheating, creating a non-linear performance drop. This is a key consideration for always-on TinyML applications.

The Roofline Model is an essential analytical tool for diagnosing utilization bottlenecks. It plots attainable performance (e.g., GOP/s) against a kernel's Operational Intensity (OI = Ops/Byte).

• Memory-Bound Region: If a layer's OI is low, performance is capped by memory bandwidth (the sloping roof). Increasing compute utilization is impossible without reducing data movement.
• Compute-Bound Region: If OI is high, performance hits the flat, horizontal roof defined by peak compute capacity. Here, compute utilization can approach 100%, but memory bandwidth is underutilized.
• Optimization Target: The goal is to shift layers into the compute-bound region via techniques like quantization (reducing byte size) and kernel fusion (increasing OI), maximizing both compute and memory efficiency.

Accurate measurement requires specialized tools that go beyond simple timing.

• Hardware Performance Counters: Access low-level MCU registers counting CPU cycles, cache misses, and instructions retired. Tools like Arm MAP or vendor-specific SDKs provide this data.
• Cycle-Accurate Simulation: For pre-silicon analysis, simulators like Gem5 or QEMU with ML extensions can estimate utilization and identify bottlenecks before hardware is available.
• Layer-Wise Profiling: Breaking down utilization by layer (e.g., using TensorFlow Lite Micro's profiler) reveals which operators are the primary consumers of compute and memory resources, guiding targeted optimization.

The maximum amount of RAM or SRAM consumed during an inference task. This includes the model's weights, activation buffers, and any intermediate tensors. In TinyML, this is often the primary constraint, as exceeding the device's SRAM causes failure. Profiling tools measure this to ensure the model fits within the hardware's memory budget.

A critical distinction in performance analysis. A workload is compute-bound when its speed is limited by the processor's arithmetic capability (e.g., MACCs/sec). It is memory-bound when limited by the speed of data movement to/from memory. TinyML models on microcontrollers with slow flash memory are often memory-bound, as fetching weights dominates the inference time. Optimizing for one state requires different techniques.

An analytical performance model that visualizes the attainable performance of a computational kernel. It plots performance (e.g., GOPs/sec) against operational intensity (Ops/Byte). The model shows two ceilings:

• A compute roof set by peak processor FLOPs.
• A memory roof set by memory bandwidth. A kernel's performance on the chart reveals if it is compute-bound, memory-bound, or optimally balanced, directly informing utilization optimization.

A key metric defined as the number of operations performed per byte of data transferred from main memory. Calculated as FLOPs / (Bytes Accessed). It quantifies the balance between computation and data movement. Kernels with low operational intensity are typically memory-bound and will have low compute utilization. Increasing operational intensity (e.g., via layer fusion or tiling) is a primary method to improve compute utilization and performance.

The detailed measurement of resource consumption for each individual layer or operator in a neural network. Essential for diagnosing utilization bottlenecks, it breaks down:

• Execution time per layer.
• Memory access patterns.
• Energy consumption. This granular data identifies specific layers that are underutilizing hardware (e.g., a low-MACC fully connected layer stalling a vector unit) or causing excessive memory traffic, guiding targeted optimizations like kernel selection or quantization.

A specialized hardware register within a processor core that counts low-level microarchitectural events. Critical for deep utilization analysis, common counters include:

• Clock cycles executed.
• Cache misses (L1, L2).
• Instructions retired.
• Stall cycles (e.g., memory wait states). Profiling tools read these counters to understand why utilization is less than 100%—for example, high cache miss rates indicate memory bottlenecks, while many stall cycles may point to data dependencies.