Inferences Per Joule (IPJ)

The inference is a single forward pass of a trained model producing a usable prediction (e.g., a classification, a regression value). For IPJ, this must be a successful inference meeting the application's accuracy threshold. Key considerations include:

• Batch Size: Typically 1 for real-time edge applications, but some accelerators benefit from micro-batching.
• Input Data Characteristics: The size and complexity of the input tensor (e.g., 96x96 RGB image vs. 1x3 accelerometer reading) directly impact compute.
• Pre/Post-Processing: Data normalization, decoding, or filtering executed on the main CPU may not be counted in core model inference time but still consumes energy.

The joule is the SI unit of energy, representing the total electrical energy consumed to perform the inference. Accurate measurement is critical and non-trivial. It encompasses:

• Active Inference Energy: Energy used by the CPU, NPU, or accelerator cores during the model's execution.
• Memory Access Energy: Significant cost from reading model weights from Flash/RAM and writing/reading intermediate activations.
• Static/Leakage Power: Baseline energy draw of the powered-on silicon, which must be apportioned to the inference timeframe.
• Peripheral & Sensor Energy: Cost of acquiring the input data (e.g., powering an image sensor or ADC) is often included in a full system-level IPJ.

IPJ requires lab-grade instrumentation for valid comparison. Common methods include:

• High-Precision Power Monitors: Devices like the Joulescope or Nordic Power Profiler Kit II, sampling at >100k samples/sec, integrated into the device's power rail.
• Software Timestamps: High-resolution timers are used to isolate the exact inference window from the power trace.
• Energy Calculation: The tool computes energy (joules) as the integral of power (watts) over the precisely measured inference time: E = ∫ P(t) dt.
• Averaging: Results are averaged over 1000s of inferences to account for system variability (cache effects, DVFS).

A crucial distinction exists in what energy is accounted for:

• Core/Accelerator IPJ: Measures only the energy of the computational unit (e.g., Ethos-U55 NPU, DSP block). This isolates hardware efficiency.
• Full System IPJ: Measures the energy of the entire device (MCU, sensors, radios in idle). This reflects real-world battery life. System IPJ is always lower than Core IPJ. Benchmarking must specify which is being reported.
• Related Metric - TOPS/W: While TOPs/W measures peak theoretical throughput, IPJ measures actual, usable work completed, making it a more practical deployment metric.

Several architectural and implementation choices negatively impact IPJ:

• High Memory Footprint: Forces frequent access to higher-energy external memory or wasteful caching.
• Inefficient Operators: Layers without optimized kernel implementations (e.g., non-fused activations) cause extra data movement.
• Poor Data Flow: Suboptimal scheduling leading to compute units waiting for data, increasing active time.
• Lack of Hardware Acceleration: Running quantized integer models on a CPU without SIMD instructions can consume 10-100x more energy than on a microNPU.
• Excessive Precision: Using 32-bit float operations where 8-bit integer (INT8) would suffice wastes energy on memory and compute.

IPJ values span orders of magnitude based on hardware and model. Examples from published research and datasheets:

• Cortex-M4F (80 MHz): ~10-100 µJ/inf for small MLP models, resulting in 10k - 100k IPJ.
• Cortex-M55 + Ethos-U55: Can achieve < 50 µJ/inf for visual wake words, yielding >20,000 IPJ.
• Ultra-Low-Power ASICs: Specialized chips for keyword spotting can achieve ~1 µJ/inf, or 1,000,000 IPJ.
• Comparison: A system with 50,000 IPJ can perform 50,000 inferences per joule. A standard 2400mAh 3.7V LiPo battery holds ~32,000 joules, theoretically enabling over 1.6 billion inferences per charge.

Milliwatt computing refers to the design paradigm for systems that operate within a power budget of milliwatts (mW), enabling battery-powered or energy-harvesting edge devices. This is the operational regime where IPJ is most critical. Key principles include:

• Ultra-low-power states: Devices spend most of their time in deep sleep modes, drawing microamps.
• Event-driven execution: The system wakes only to process sensor triggers or scheduled inferences.
• Energy-proportional computing: The energy consumed is directly proportional to the useful computational work performed, a core goal reflected in a high IPJ score.

Hardware-Aware Neural Architecture Search (HW-NAS) is an automated process for designing neural networks that directly optimizes for hardware performance metrics like latency, energy consumption, and memory usage. Unlike standard NAS which focuses only on accuracy, HW-NAS uses the target hardware (e.g., an ARM Cortex-M4) as a feedback loop during the search. It is a primary methodology for architecting models that achieve high Inferences Per Joule (IPJ) by co-designing the algorithm and the silicon it runs on.

Model quantization is a compression technique that reduces the numerical precision of a neural network's weights and activations. Converting parameters from 32-bit floating-point to 8-bit integers (INT8) or lower is standard. This directly improves IPJ by:

• Reducing memory bandwidth: Less data is moved between memory and compute units, a major energy cost.
• Enabling integer arithmetic: Integer operations are faster and more energy-efficient than floating-point on most microcontrollers.
• Decreasing model size: Allows larger models to fit in on-chip SRAM, avoiding power-hungry external flash accesses.

Worst-Case Execution Time (WCET) is the maximum possible time a specific task, such as a single model inference, could take to complete on an edge device. For battery-powered systems, energy is the integral of power over time. Therefore, a bounded, predictable WCET is essential for calculating and guaranteeing a minimum Inferences Per Joule (IPJ). Techniques to ensure a tight WCET include:

• Static memory allocation to prevent garbage collection pauses.
• Using deterministic kernels (e.g., CMSIS-NN).
• Avoiding data-dependent control flows where possible.

The Arm Ethos-U55 is a micro Neural Processing Unit (microNPU), a dedicated hardware accelerator designed as a coprocessor for Cortex-M systems. It dramatically improves Inferences Per Joule (IPJ) by:

• Specialized datapaths: Efficiently executes the tensor operations (convolutions, fully connected layers) common in neural networks.
• Weight encoding: Uses advanced compression like weight pruning and shared exponents to reduce memory traffic.
• Direct SRAM access: Minimizes energy spent on data movement. A system using an Ethos-U55 can achieve IPJ figures orders of magnitude higher than CPU-only inference.

A Once-for-All network is a large, trainable supernet containing a vast number of smaller sub-networks within its weight-sharing structure. It is trained once. Then, for deployment, an efficient sub-network is extracted to match the exact latency, memory, and energy constraints (and thus IPJ target) of a specific edge device—all without retraining. This enables a single model to serve a heterogeneous fleet of devices, each running a subnet optimized for its own hardware capabilities and power budget.