Energy per Inference

This is the energy consumed by the processor's transistors actively switching during computation. It is the primary component for the inference task itself and is directly proportional to:

• Clock Frequency: Higher frequency increases dynamic power quadratically (P ~ CV²f).
• Supply Voltage: Often scaled with frequency (DVFS). Lower voltage saves significant power.
• Computational Workload: The total number of operations, measured in MACC count or FLOPs.
• Hardware Efficiency: How well the processor's architecture (e.g., a Neural Processing Unit) maps to the model's operations.

Example: A 100 MHz ARM Cortex-M4 executing a dense layer will consume more dynamic energy than a specialized NPU executing the same layer at a lower voltage and frequency.

This is the constant energy dissipated due to leakage currents when the chip is powered on, even if idle. It becomes a dominant factor in low-duty-cycle applications where the device spends most of its time sleeping.

• Technology Node: Smaller nanometer processes have higher leakage per transistor.
• Temperature: Leakage current increases exponentially with junction temperature.
• Inference Time: Static energy = (Static Power) × (Inference Latency). A slower, more efficient inference may paradoxically increase static energy if latency increases significantly.

For always-on sensing with infrequent inferences, optimizing the system's sleep current and wake-up time is often more critical than optimizing dynamic compute energy.

Moving data is often more expensive than computing on it. Energy per inference is heavily influenced by memory hierarchy accesses:

• SRAM vs. Flash Access: Reading a 32-bit word from on-chip SRAM can consume ~100 pJ, while accessing external Flash can consume > 1000 pJ.
• Model Activation Movement: The energy cost of reading/writing intermediate feature maps (activations) between layers can exceed the cost of the layer's computations, especially in memory-bound layers.
• Weight Fetching: Quantized 8-bit weights stored in Flash consume less energy to fetch than 32-bit floats. Techniques like weight pruning reduce the number of fetches.

Architectural optimizations focus on maximizing data reuse in fast, low-energy caches/SRAM to minimize costly accesses to main memory or Flash.

The energy cost of acquiring and preprocessing the input data is frequently overlooked but can dwarf the inference cost.

• Sensor Wake-up & Sampling: A MEMS microphone or accelerometer may consume milliwatts during active sampling.
• Analog-to-Digital Converter (ADC): High-resolution or high-speed ADC conversion is power-intensive.
• Preprocessing Pipeline: Operations like audio FFT, image debayering, or sensor fusion on a general-purpose core add computational overhead.

Total End-to-End Latency directly impacts this component. An efficient system uses low-power sensor front-ends and dedicated, minimal preprocessing to provide the model with ready-to-use data.

The system's ability to aggressively power-gate unused components and rapidly transition between sleep and active states is paramount for energy efficiency.

• Power Gating: Turning off power to entire blocks (e.g., a peripheral, a core, an NPU) when not in use.
• Clock Gating: Halting the clock to idle logic, eliminating dynamic power for that block.
• Wake-up Time & Overhead: The energy and time penalty of transitioning from sleep to full operational mode. Frequent, short inferences favor low wake-up overhead.
• Deterministic Execution: Predictable inference latency allows the system scheduler to return to deep sleep immediately, minimizing idle time.

Effective management turns a high peak power inference into a low average power system.

Accurately measuring energy per inference requires specialized tools and a rigorous methodology.

• High-Resolution Power Monitors: Devices like the Joulescope or Otii Arc that sample current at > 1 MS/s to capture microsecond-scale power transients.
• Synchronized Triggering: The measurement must be precisely triggered to start/stop with the inference task, isolating it from system idle or other tasks.
• Layer-wise Energy Profiling: Tools like the MLPerf Tiny Energy Measurement Kit break down energy consumption per network layer, identifying hotspots.
• Hardware-in-the-Loop (HIL) Testing: Running the actual compiled firmware on the target MCU while measuring power provides the only ground-truth measurement, accounting for all compiler optimizations and hardware effects.

Benchmarks like TinyMLPerf standardize these measurement procedures for fair comparison.

The total time delay from input to prediction output for a single inference. For real-time systems, latency is often the primary constraint, directly trading off with energy consumption. Lower latency typically requires higher clock speeds or parallel compute, increasing dynamic power. Key considerations:

• Measured in milliseconds (ms) or microseconds (µs).
• Includes model execution plus any pre/post-processing.
• Must be analyzed alongside energy to understand the latency-energy Pareto frontier.

The maximum RAM/SRAM consumed during inference, including model weights, activations, and intermediate buffers. Memory is a severe bottleneck on microcontrollers (often < 512KB). High memory usage forces frequent accesses to slower, higher-energy external memory, drastically increasing energy per inference. Key aspects:

• SRAM access consumes significantly more power than compute on many MCUs.
• Techniques like activation spilling trade compute for memory.
• Profiling tools measure layer-by-layer memory peaks to identify optimization targets.

The total number of multiply-accumulate operations required for a forward pass. MACCs are the fundamental compute workload for convolutional and fully connected layers. While a proxy for complexity, the energy per MACC varies drastically based on:

• Data precision (8-bit vs. 32-bit).
• Hardware support (dedicated MAC unit vs. software emulation).
• Data reuse and caching efficiency. A low MACC count model (e.g., MobileNet) is necessary but not sufficient for low energy; inefficient memory access can dominate power.

The two components of total power dissipation. Dynamic power is consumed during transistor switching (computation and memory access) and scales with activity, clock frequency, and voltage squared. Static power (leakage) is constant when the chip is powered. For TinyML:

• Dynamic power dominates during active inference.
• Static power becomes significant in always-on, duty-cycled systems where the device spends most time idle.
• Minimizing energy requires optimizing both: reducing active time (dynamic) and enabling deep sleep states (static).

An analytical performance model that visualizes the attainable performance of a computational kernel. It plots operations per second (e.g., GOP/s) against operational intensity (OPs/byte). The model reveals if a layer or model is:

• Compute-bound: Performance limited by the chip's peak compute rate.
• Memory-bound: Performance limited by memory bandwidth. For energy analysis, a memory-bound kernel spends most of its time and energy on data movement, not arithmetic. Optimizations must target the bottleneck: improving data reuse (compute-bound) or reducing footprint (memory-bound).