Layer-wise Profiling

Layer latency is the time, typically measured in microseconds or milliseconds, required to execute a single forward pass of a specific neural network layer (e.g., Conv2D, DepthwiseConv2D, FullyConnected). It is the primary metric for identifying computational bottlenecks.

• Measurement: Captured via hardware timers or cycle-accurate simulators.
• Critical for: Real-time systems where meeting inference deadlines is mandatory.
• Example: A 3x3 depthwise convolutional layer might consume 15 ms on a target MCU, while a subsequent pointwise convolution takes only 2 ms, highlighting the first as the primary target for kernel optimization or algorithmic substitution.

The activation memory footprint is the peak amount of RAM (SRAM) consumed by the intermediate output tensors (activations) of a specific layer during its execution. This is often the dominant factor in a model's total RAM requirement.

• Defined by: The output tensor's dimensions (batch, height, width, channels) and numerical precision (e.g., int8, float32).
• Impact: Exceeding available SRAM forces spilling to slower flash memory, drastically increasing latency and energy use.
• Optimization Target: Techniques like activation quantization, layer fusion, and operator tiling directly aim to reduce this per-layer footprint.

Weight memory access refers to the volume and pattern of data movement required to read a layer's learned parameters (weights and biases) from storage (typically Flash) into compute-ready memory (SRAM).

• Cost Dominance: On microcontrollers, energy consumed by reading memory (~1-10 pJ/byte) can far exceed the energy of the actual computation (~0.1-1 pJ/OP).
• Profiling Insight: Measures bytes read per layer. Large fully-connected or convolutional layers with many weights are major contributors.
• Mitigation: Weight pruning and structured sparsity reduce the number of weights that must be fetched, while caching strategies optimize reuse.

The MAC operation count is the total number of multiply-accumulate computations performed within a specific layer. It is a fundamental measure of computational workload and is strongly correlated with latency and energy consumption.

• Calculation: For a Conv2D layer: MACs = K_h * K_w * C_in * C_out * H_out * W_out.
• Use Case: Used in roofline model analysis to determine if a layer is compute-bound (high operational intensity) or memory-bound (low operational intensity).
• TinyML Context: A high MAC count layer may be a candidate for replacement with a factorized version (e.g., replacing a standard convolution with a depthwise separable convolution).

Energy consumption per layer is the total electrical energy, measured in microjoules (µJ), dissipated by the hardware while executing a specific layer. It is the product of average power draw and layer latency.

• Components: Includes dynamic energy (from transistor switching during computation and memory access) and static energy (leakage current).
• Measurement: Requires specialized equipment (e.g., precision power monitors like Joulescope) or integrated energy counters on advanced MCUs.
• Optimization Goal: The primary metric for battery-powered applications. Profiling reveals which layers are the most energy-hungry, guiding optimizations like precision reduction or hardware-offloading to a more efficient NPU block.

Hardware utilization measures the efficiency with which a layer's computation utilizes the available hardware resources, such as the processor's DSP units, NPU matrix multipliers, or memory bandwidth.

• Low Utilization Causes: Inefficient kernel implementations, data dependencies that cause pipeline stalls, or mismatched data layouts leading to excessive cache misses.
• Profiling Method: Accessed via hardware performance counters that track events like cycles stalled waiting for memory, DSP unit idle cycles, or cache hit/miss rates.
• Actionable Insight: A layer with low DSP utilization but high latency is likely memory-bound, indicating a need to optimize data layout or employ direct memory access (DMA) for better bandwidth.

Inference latency is the total time delay from input presentation to prediction output for a single model inference. In layer-wise analysis, this is the sum of the latencies of all individual layers. It is a critical metric for real-time applications, where exceeding a latency budget can cause system failure. Profiling helps identify which specific layers (e.g., a large dense layer or a depthwise convolution) are the primary contributors to the total delay.

Peak memory usage is the maximum amount of RAM consumed during inference, encompassing model weights, layer activations, and intermediate buffers. Layer-wise profiling breaks this down to show the memory footprint of each layer's output tensor, which is crucial for microcontrollers with kilobytes of SRAM. Understanding this profile allows engineers to optimize data placement, implement layer fusion to reuse buffers, and avoid out-of-memory errors.

Energy per inference measures the total electrical energy (in microjoules) consumed to complete one forward pass. Layer-wise energy profiling correlates computational and memory activity of each operator with hardware power states. Energy-intensive layers often involve high Multiply-Accumulate (MACC) counts or frequent off-chip memory accesses. This data is essential for designing battery-powered devices, enabling targeted optimizations like approximating expensive layers or adjusting clock speeds.

MACC (Multiply-Accumulate) count is the total number of fused multiplication and addition operations required by a model. It is a primary proxy for computational workload. Layer-wise profiling assigns a MACC count to each layer (e.g., a CONV2D layer's MACCs = H_out * W_out * C_out * K_h * K_w * C_in). This count, combined with hardware performance data, helps predict if a layer is compute-bound (limited by arithmetic units) and is a key input for performance models like the Roofline Model.

The Roofline Model is an analytical performance model that visualizes attainable performance as a function of operational intensity (operations per byte of DRAM access). In layer-wise analysis, each layer can be plotted as a point on the roofline chart. Layers below the memory-bound roof are limited by data movement bandwidth, while those below the compute-bound roof are limited by arithmetic throughput. This model guides optimization strategy: for memory-bound layers, focus on data reuse and cache blocking; for compute-bound layers, focus on kernel efficiency and parallelization.

A profiling tool is software that instruments a system to collect detailed runtime metrics. For TinyML layer-wise profiling, tools like TensorFlow Lite Micro's profiler, Arm Keil MDK Profiler, or Percepio Tracealyzer intercept function calls to record timestamps, memory allocations, and hardware performance counter events (e.g., cache misses, cycles stalled). These tools transform raw telemetry into the layer-by-layer breakdowns that drive optimization decisions, bridging the gap between the model graph and the hardware's actual behavior.