Frames Per Joule (FPJ)

The Frames Per Second (FPS) component measures the system's processing speed. It is the average number of image frames the vision pipeline (sensor capture, preprocessing, neural network inference, post-processing) can complete in one second under a sustained workload.

• Measured At: The output of the final processing stage (e.g., after bounding box generation or classification).
• Critical Dependency: Heavily influenced by model architecture complexity, hardware accelerator throughput (e.g., TOPS of an NPU), and memory bandwidth.
• Benchmarking: Must be measured using a representative, continuous stream of input data, not a single batch, to account for system overhead.

The Average Active Power component, measured in watts, represents the total electrical energy drawn per second by the relevant subsystems during inference. This is not peak power but the steady-state consumption.

• Measurement Scope: Includes the AI accelerator (CPU, GPU, NPU), associated memory, and any dedicated vision pre-processing units. Often excludes peripherals like displays.
• Tooling: Measured using precision external power monitors (e.g., Monsoon power meter, Joulescope) or internal PMU (Power Management Unit) telemetry.
• Key Insight: Dynamic power (from switching activity) is dominant during active computation, but static (leakage) power becomes significant in advanced process nodes.

Frames Per Joule is calculated by dividing the sustained throughput by the average power draw during that same measurement period.

Formula: FPJ = FPS / Average Power (Watts)

• Derivation: Since 1 Watt = 1 Joule/second, dividing Frames/Second by Joules/Second cancels the time unit, leaving Frames/Joule.
• Interpretation: A score of 50 FPJ means the system processes 50 frames for every joule of energy consumed. A higher FPJ indicates greater energy efficiency.
• Example: A system achieving 30 FPS while drawing 0.5 Watts has an FPJ of 30 / 0.5 = 60.

FPJ can be scoped differently, leading to distinct optimization insights.

• System-Level FPJ: Measures the entire application's efficiency, including OS overhead, sensor power, and memory traffic. This is the most relevant metric for product architects and battery life estimation.
• Accelerator-Only FPJ: Measures only the AI accelerator's efficiency (e.g., NPU core). This isolates hardware/compiler performance and is used by silicon architects and kernel developers.
• Reporting Requirement: Any FPJ benchmark must explicitly state its measurement scope to be meaningful.

FPJ and latency (frame processing time) are related but distinct. Optimizing for one can negatively impact the other.

• High FPJ ≠ Low Latency: A system can be batch-processing frames efficiently (high FPJ) but have high per-frame latency. For real-time applications, both metrics must be evaluated.
• The Batched Processing Advantage: Processing frames in batches often dramatically improves FPJ by amortizing fixed overheads, but increases latency. The optimal batch size is a key trade-off.
• Use Case Dictates Priority: A security camera analyzing recorded footage prioritizes FPJ; an autonomous drone avoiding obstacles prioritizes latency, then FPJ.

Accurate FPJ measurement requires controlled, reproducible methodology to be a valid comparison tool.

• Stable Thermal State: Hardware must reach a steady thermal state before measurement, as performance and power vary with temperature.
• Fixed Workload: Use a standard, representative dataset (e.g., COCO validation set for object detection) at a fixed resolution and batch size.
• Measurement Duration: Capture data over a sufficiently long period (e.g., 2-5 minutes) to average out periodic OS tasks and thermal fluctuations.
• Tool Example: MLPerf Tiny Inference benchmark provides a standardized framework for measuring FPJ and other metrics on edge devices.

Reducing a model's computational and memory footprint directly lowers the energy per inference. Post-training quantization converts model weights and activations from 32-bit floating-point to 8-bit integers (INT8), drastically reducing the energy cost of memory accesses and arithmetic operations. More aggressive techniques like weight pruning remove redundant parameters, creating sparse models that skip computations, while knowledge distillation trains a compact 'student' model to mimic a larger 'teacher'.

Efficient execution requires software kernels tailored to the target silicon. This involves:

• Utilizing low-level intrinsics and assembly for critical loops on CPUs/GPUs.
• Leveraging specialized instructions for matrix multiplication (e.g., ARM SME) or dot products.
• Implementing sparse compute kernels that skip multiplications with zero-valued weights from pruning.
• Employing winograd or FFT-based convolution algorithms that reduce the total operation count for specific layer types.

DVFS is a foundational power management technique. The processor's operating voltage and clock frequency are dynamically scaled based on the real-time computational demand of the inference workload. Running at a lower frequency/voltage quadratically reduces dynamic power consumption. Effective use requires profiling the model to find the sweet spot where the system is just fast enough to meet latency requirements without wasting energy on excess performance headroom.

This technique eliminates leakage power in idle hardware blocks. Fine-grained power gating shuts off power to unused cores, caches, or accelerators within the SoC. Wake-on-inference architectures use a tiny, always-on microcontroller or microNPU to run a minimal detection model (e.g., for keyword spotting or motion detection). Only when a trigger event occurs is the main, high-power AI accelerator powered on to run the full vision model, dramatically improving system-level FPJ.

Avoiding continuous processing is key for sensor-based systems. Instead of analyzing every frame at a fixed rate, inference is triggered only by meaningful events:

• A hardware interrupt from a passive infrared (PIR) motion sensor.
• Software detection of significant change between consecutive image frames. The system then operates with a high duty cycle, spending most of its time in a deep sleep state and waking briefly to process data, minimizing average power.

Optimizing the model's execution plan saves energy by reducing data movement, which is often more costly than computation itself. Techniques include:

• Operator fusion: Combining consecutive layers (e.g., Conv + BatchNorm + ReLU) into a single kernel to avoid writing intermediate tensors to slow DRAM.
• Efficient memory layout: Using NHWC vs. NCHW formats to optimize for cache locality.
• Static memory planning: Pre-allocating all buffers at load time to eliminate runtime allocation overhead and fragmentation.

Performance-per-watt is a foundational system efficiency metric, defined as the amount of useful computational work delivered per watt of power consumed. It provides a high-level view of a hardware platform's efficiency.

• Broader Scope: While FPJ measures a specific vision task, performance-per-watt can apply to any workload (e.g., inferences/sec/W, GFLOPs/W).
• Design Target: It is a primary figure of merit for processors in battery-constrained devices, directly influencing product battery life and thermal design.
• Trade-off Analysis: Engineers use this metric to evaluate the efficiency trade-offs between different chips (e.g., CPU vs. GPU vs. NPU) for a given AI model.

Joule per inference is the inverse and more granular counterpart to FPJ. It measures the absolute energy, in joules, required to perform a single forward pass of a machine learning model.

• Direct Measurement: Provides a concrete, absolute energy cost for one unit of work, making it easier to project total battery drain from a known inference rate.
• Calculation: FPJ can be derived as 1 / (Joule per Inference). If a model consumes 0.02 J per inference, its FPJ is 50.
• Use Case: Critical for calculating the total energy budget of an application with a fixed number of required inferences.

Dynamic Voltage and Frequency Scaling (DVFS) is a hardware power management technique that dynamically adjusts a processor's operating voltage and clock frequency in response to real-time computational demand.

• Energy Savings: Since dynamic power is proportional to the square of the voltage (P ∝ V² * f), reducing voltage for lower workloads saves significant energy.
• FPJ Impact: An optimized DVFS policy can dramatically increase a system's FPJ by running the processor at the most energy-efficient operating point for a given inference task.
• System Control: Modern operating systems and firmware use DVFS governors to balance performance and power based on workload queues.

Power gating is a circuit-level technique that completely shuts off power supply to inactive or idle blocks of a silicon chip using header or footer sleep transistors.

• Eliminates Leakage: Unlike clock gating, it stops both dynamic power and static leakage power in the gated region, which is crucial for modern nanometer processes where leakage is significant.
• Granularity: Applied at the level of cores, cache banks, or specific functional units (like an unused NPU block).
• Wake-up Latency: Introducing power gating adds a time and energy penalty to turn the block back on, which must be amortized over a sufficiently long sleep period.

Wake-on-inference is a system architecture pattern where a tiny, ultra-low-power always-on coprocessor monitors sensor data and only activates the main, high-performance AI accelerator when a specific trigger is detected.

• Hierarchical Sensing: The always-on core runs a very small, efficient model (e.g., for keyword spotting or simple motion detection). Upon detecting an event, it wakes the main system to run a larger, more accurate model.
• Massive Energy Savings: The main accelerator, which consumes milliwatts or watts, remains in a deep sleep state until absolutely needed, drastically improving overall system FPJ.
• Use Case: Found in smart speakers (wake word detection), smartphones (raise-to-wake), and security cameras.

A milliwatt budget is a strict power consumption constraint, typically in the single-digit to tens of milliwatts range, imposed on an edge device or subsystem to ensure operation within the energy limits of a small battery or energy harvester.

• Design Imperative: This is not a target but a hard limit. The entire inference pipeline—sensor, memory, processor—must operate below this average power ceiling.
• Drives Architecture Choice: A milliwatt budget often forces the use of microcontrollers, extreme model compression (e.g., 8-bit or lower quantization), and aggressive duty cycling.
• FPJ Context: The FPJ metric is used to calculate how many frames can be processed within this fixed power envelope over time (Frames = FPJ * (Budget in Watts) * Time in Seconds).