Inference-Per-Watt

The foundational hardware metric is TOPS per Watt (Trillions of Operations Per Second per Watt). This measures the raw computational efficiency of the AI accelerator (e.g., NPU, GPU, or microcontroller). Key factors include:

• Architecture: Dedicated matrix multiplication units (MACs) vs. general-purpose cores.
• Process Node: Smaller semiconductor nodes (e.g., 5nm) typically offer better efficiency.
• Memory Hierarchy: Energy cost of accessing SRAM vs. DRAM; in-memory computing architectures minimize data movement.
• Voltage/Frequency Scaling: Operating in a near-threshold voltage regime can dramatically reduce dynamic power.

The neural network's architecture directly dictates the number and type of operations required. Efficient models maximize accuracy per operation (FLOPS). Key techniques include:

• Model Compression: Quantization (e.g., INT8 vs. FP32), pruning, and knowledge distillation reduce computational workload.
• Efficient Architectures: Use of depthwise separable convolutions (MobileNet), attention mechanisms optimized for edge (MobileViT).
• Sparsity: Leveraging activation and weight sparsity to skip zero-operations.
• Early Exits: Allowing simple inputs to exit the network at earlier layers, bypassing heavy computation.

The runtime environment and power management subsystems determine the actual energy drawn during inference. This involves dynamic control of non-compute components.

• Dynamic Voltage and Frequency Scaling (DVFS): Adjusts processor voltage/clock to match the inference workload.
• Power Gating & Clock Gating: Shuts off power or clocks to idle subsystems (e.g., unused cores, peripherals).
• Memory Power States: Placing DRAM in low-power self-refresh modes between inference batches.
• Wake-on-Event: Using an ultra-low-power Always-On (AON) domain to wake the main processor only when a trigger occurs.

The compiler and runtime libraries translate the model into hardware-executable instructions. Their efficiency determines how close real performance gets to peak TOPS/W.

• Kernel Fusion: Combining multiple neural network operations into a single, optimized kernel to reduce intermediate data writes to memory.
• Operator-Level Tiling: Strategically partitioning computations to fit within fast, on-chip SRAM/cache.
• Fixed-Point Arithmetic: Using optimized integer math libraries instead of floating-point emulation on microcontrollers.
• Scheduling: Power-aware scheduling of parallel operations to minimize contention and peak current draw.

Inference-per-watt is not an absolute number; it is defined by the specific benchmark conditions. Comparing metrics requires strict contextual alignment.

• Benchmark Model: Standard models (e.g., MobileNet-V1, ResNet-50) or proprietary workloads.
• Precision: Stated for a specific data type (FP32, FP16, INT8).
• Batch Size: Throughput (inferences/sec) and power scale non-linearly with batch size. Batch size=1 is critical for real-time edge scenarios.
• Power Measurement Point: Chip power (core only) vs. total board power (including RAM, power conversion losses).
• Temperature & Voltage: Performance and leakage power are temperature-dependent.

The ultimate system design lever is trading prediction quality for energy savings. Inference-per-watt must be evaluated alongside a target accuracy.

• Pareto Frontier: The curve plotting accuracy vs. energy per inference. Optimal designs lie on this frontier.
• Adaptive Inference: Dynamically selecting model precision or architecture based on input difficulty or battery state.
• Approximate Computing: Using inherently approximate hardware or skipping operations in error-resilient applications.
• End-to-End Optimization: Co-designing the model, software, and hardware to push the Pareto frontier, maximizing inferences per watt for a required accuracy threshold.

Reducing a model's memory footprint and computational cost directly lowers the energy required per inference. Post-training quantization converts model weights and activations from 32-bit floating-point to lower precision formats like 8-bit integers (INT8) or even binary values, drastically cutting the energy of arithmetic operations. Weight pruning removes redundant or insignificant connections from the neural network, creating a sparse model that requires fewer computations. Knowledge distillation trains a smaller, more efficient 'student' model to mimic the behavior of a larger 'teacher' model, preserving accuracy with a fraction of the parameters.

This automated process discovers neural network architectures optimized for a specific microcontroller's constraints. Instead of designing networks for general-purpose GPUs, HW-NAS uses search algorithms to evaluate candidate architectures against a hardware cost model that includes:

• Energy per inference
• Peak memory usage
• Latency The resulting networks use operations and layer dimensions that maximize efficiency on the target hardware, such as preferring depthwise separable convolutions over standard convolutions for vision tasks on ARM Cortex-M cores.

DVFS is a foundational system-level technique that dynamically adjusts a processor's operating voltage and clock frequency in response to real-time computational demand. For ML inference, the system can:

• Down-clock the CPU or accelerator during less intensive layers of a network.
• Ramp up performance only for computationally heavy layers (e.g., large matrix multiplications). Since dynamic power scales with the square of the voltage and linearly with frequency (P ∝ CV²f), even small reductions can yield significant energy savings. This requires tight integration between the inference runtime and the operating system's power governor.

These algorithmic techniques avoid unnecessary computations. Activation sparsity leverages the fact that many neuron outputs (e.g., after a ReLU function) are zero; specialized hardware or software can skip computations involving these zeros. Early exit networks (or multi-exit networks) place intermediate classifiers at shallower layers within the model. For 'easy' input data that meets a confidence threshold at an early exit, the inference terminates, bypassing the remaining, more computationally expensive layers. This creates a dynamic compute graph where energy consumption adapts to input complexity.

These are circuit-level techniques to eliminate different types of power waste. Power gating uses header or footer switches to completely cut off the power supply (VDD) to idle hardware blocks, such as a dedicated AI accelerator or a peripheral. This eliminates both dynamic and static (leakage) power consumption for that block. Clock gating disables the clock signal to circuit regions that are not currently performing useful work, preventing transistors from switching and thus eliminating dynamic power dissipation. In MCUs, entire processor cores or SRAM banks can be power- or clock-gated between inference batches.

These architectures attack the dominant energy cost in modern computing: data movement. In-Memory Computing (IMC) performs computation directly within the memory array (e.g., SRAM, RRAM) using analog or digital techniques, avoiding the energy-intensive transfer of weights and activations to a separate processing unit. Near-Memory Computing places lightweight processing units close to memory banks to minimize data movement distance. For TinyML, this can manifest as specialized machine learning accelerators with tightly coupled weight buffers and systolic arrays that keep data flow localized and energy-efficient.

A combined metric, calculated as energy consumed multiplied by execution time, used to evaluate the fundamental trade-off between the performance and energy efficiency of a computing system or algorithm. For TinyML, a lower EDP indicates a more optimal balance for battery-powered, latency-sensitive applications.

• Key Insight: Optimizing solely for low energy may result in unacceptable latency. EDP forces consideration of both.
• Application: Used to compare different neural network architectures, quantization schemes, or hardware accelerators for a given task.

The fundamental design space in machine learning systems where reducing computational cost saves energy but may decrease prediction accuracy. Key levers for managing this trade-off in TinyML include:

• Quantization: Reducing numerical precision (e.g., 32-bit float to 8-bit integer) saves memory bandwidth and compute energy.
• Pruning: Removing insignificant weights or neurons reduces model size and operations.
• Architectural Choices: Using shallower networks or early exit networks.
• Dynamic Adjustment: Systems can adapt precision or model complexity based on available battery or task criticality.

A core hardware power management technique that dynamically adjusts a processor's operating voltage and clock frequency based on real-time computational workload. It is critical for optimizing Inference-Per-Watt.

• Mechanism: Lowering voltage and frequency reduces dynamic power consumption quadratically and linearly, respectively.
• TinyML Application: An inference engine can request a lower frequency/voltage for a lightweight model and a higher one for a complex model, matching power to need.
• Integration: Managed by the OS or runtime via standards like the Advanced Configuration and Power Interface (ACPI).

A circuit-level technique that completely cuts off the power supply to idle logic blocks using header or footer switches. This eliminates both dynamic and static (leakage) power consumption, which is dominant in advanced silicon nodes.

• Contrast with Clock Gating: Clock gating stops clock toggles (saving dynamic power); power gating removes all power (saving dynamic + leakage).
• TinyML Use Case: A dedicated neural processing unit (NPU) block can be power-gated between inference bursts, while a tiny always-on (AON) domain handles sensor monitoring.
• Overhead: Involves a latency and energy cost to save/restore state when waking the block.

An aggressive circuit design paradigm where digital logic operates with a supply voltage close to the transistor's threshold voltage. This achieves dramatic energy savings (5-10x) at the cost of significantly reduced performance and increased susceptibility to noise and process variations.

• Trade-off: Excellent for ultra-low-power, non-time-critical background processing in always-on contexts.
• Relation to Inference-Per-Watt: Enables extremely efficient inferences for simple models where latency is not critical.
• Challenge: Requires specialized design and error-tolerant architectures, linking to approximate computing principles.

A system power management feature enabling ultra-low-power always-on sensing. The main processor sleeps in a deep low-power state until a specific external trigger awakens it.

• Mechanism: A simple, low-power circuit (in the AON domain) continuously monitors a sensor or a wake-up radio. Upon detecting a predefined pattern or threshold, it triggers a full system wake-up.

• Impact on Inference-Per-Watt: Dramatically improves system-level efficiency by ensuring the high-power ML accelerator only runs when necessary. The overall system's effective 'inferences per joule' is vastly improved.

• Example: An acoustic event detection model running on a low-power MCU in the AON domain wakes the main NPU only when it detects a keyword.