Energy per Inference

The most direct driver of energy consumption is the sheer number of multiply-accumulate (MAC) operations required. A model with 100 million parameters demands significantly more data movement and computation than one with 10 million.

• Memory Access Cost: Fetching weights from DRAM often consumes more energy than the computation itself.
• Sparsity: Models with many zero-valued weights can skip computations, but only if the hardware supports unstructured sparsity.
• Example: A ResNet-50 requires ~4 GFLOPs per inference, while a MobileNetV3 requires only ~0.2 GFLOPs, directly correlating to a lower mJ per inference.

The bit-width of weights and activations has a quadratic impact on energy. Moving from FP32 (32-bit) to INT8 (8-bit) reduces the energy per operation by an order of magnitude.

• FP32: Standard training precision, high energy cost.
• FP16/BF16: Halves memory bandwidth and compute energy on modern GPUs and NPUs.
• INT8: The sweet spot for edge inference, supported by TensorRT and OpenVINO via post-training quantization.
• Extreme Low-Bit: Research into 4-bit and binary neural networks pushes energy to the absolute minimum, often requiring quantization-aware training to retain diagnostic accuracy.

General-purpose CPUs are inefficient for the parallelizable tensor operations of deep learning. Dedicated accelerators use specialized silicon to maximize performance per watt.

• GPU: Massively parallel, high throughput, but higher idle power. Suitable for complex gigapixel inference.
• NPU (Neural Processing Unit): A dedicated inference accelerator integrated into SoCs like Jetson Orin, offering the best energy efficiency for sustained inference.
• FPGA: Allows for custom, dataflow-optimized pipelines with deterministic, ultra-low latency and power, ideal for scanner-side AI.
• ASIC: A custom chip (e.g., Google's TPU Edge) provides the theoretical maximum efficiency for a fixed model.

Processing multiple inputs simultaneously amortizes the cost of loading model weights from memory. A larger batch size increases total throughput but also increases latency and peak power draw.

• Single-Batch Latency: Critical for real-time scanner-side applications where an image must be processed immediately.
• Energy Efficiency: A batch size of 1 is the least energy-efficient per inference because weight loading overhead is not shared.
• Thermal Constraints: Edge devices often throttle performance under sustained high batch loads due to heat buildup, increasing total energy per inference over time.

The design of the neural network's computational graph fundamentally dictates the number of operations required. Efficient architectures are purpose-built to minimize FLOPs without sacrificing accuracy.

• Depthwise Separable Convolutions: Used in MobileNet, these factor a standard convolution into two cheaper operations, drastically reducing computation.
• Vision Transformers (ViTs): While powerful, their self-attention mechanism scales quadratically with sequence length, potentially increasing energy for high-resolution inputs.
• Neural Architecture Search (NAS): An automated process that discovers highly efficient, hardware-aware model architectures optimized for a specific energy budget.

The dynamic power consumption of a CMOS circuit is proportional to the square of the voltage and linearly to the clock frequency. Dynamic Voltage and Frequency Scaling (DVFS) is a critical firmware-level lever.

• Race-to-Idle: Running a core at maximum frequency to finish a task quickly, then entering a deep sleep state, can be more efficient than running slowly for a long duration.
• Undervolting: Reducing the supply voltage to the minimum stable level for a given frequency can yield quadratic energy savings.
• Thermal Throttling: As a chip heats up, it may automatically reduce frequency, paradoxically increasing the time to complete an inference and potentially increasing total energy consumed.

A model optimization paradigm that incorporates the specific constraints of a target accelerator directly into the neural network training process. By simulating latency and energy costs during training, the optimizer searches for architectures that minimize millijoules per forward pass.

• Uses differentiable energy models of the target silicon
• Produces Pareto-optimal models balancing accuracy and power
• Critical for custom ASIC and FPGA deployments

The use of a reconfigurable Field-Programmable Gate Array to implement custom, highly parallel hardware circuits for neural network inference. FPGAs offer deterministic, ultra-low energy per inference by eliminating the overhead of instruction fetch and decode present in general-purpose processors.

• Sub-millijoule inference achievable for quantized models
• Ideal for streaming data from medical sensors
• Enables hard real-time diagnostic pipelines

An execution strategy that partitions a single AI workload across different processors on a System-on-a-Chip. By mapping specific layers to the most efficient core, it minimizes total energy per inference.

• Lightweight pre-processing on a low-power DSP
• Convolutional layers accelerated on an NPU
• Control logic handled by a CPU core
• Maximizes battery life in handheld diagnostic tools

A technique that uses lower-precision arithmetic like FP16 or INT8 for the majority of a model's operations while retaining FP32 for sensitive layers. This balances speed improvements with minimal accuracy loss, directly reducing the energy per inference without requiring full model retraining.

• Automatic mixed precision (AMP) in modern frameworks
• Reduces memory bandwidth pressure
• Standard practice for deploying large vision models at the edge