Model Efficiency

Inference latency is the total time delay from input presentation to prediction output for a single inference. It is the primary determinant of system responsiveness and is critical for real-time applications.

• Measured in milliseconds (ms) or microseconds (µs) on MCUs.
• Directly impacts user experience and control loop stability.
• Factors include model architecture, hardware clock speed, memory bandwidth, and operator kernel efficiency.
• Example: An audio keyword spotting model must have latency < 100 ms to feel instantaneous.

Peak memory usage is the maximum amount of RAM (typically SRAM) consumed during inference. This includes model weights, activations, and intermediate tensor buffers. It is the most stringent constraint for MCU deployment.

• Measured in kilobytes (KB).
• Must be less than the total available SRAM on the target device.
• Activation memory often dominates, not the static weight storage.
• Exceeding available memory causes system crashes or reliance on slower external memory.

Energy per inference is the total electrical energy consumed to complete a single forward pass. It is the product of average power and inference time, determining battery life in always-on applications.

• Measured in microjoules (µJ) or millijoules (mJ).
• Dynamic energy from computation and data movement is the primary component.
• Static leakage energy becomes significant in low-duty-cycle systems.
• Example: A wildlife monitoring camera must operate for months on a small battery, requiring < 10 mJ per inference.

The Multiply-Accumulate (MACC) count (or FLOP count) quantifies the computational workload of a model. One MACC is typically counted as two floating-point operations (FLOPs). It is a hardware-agnostic measure of complexity.

• Reported in millions (MMAC) or billions (GMAC).
• Directly correlates with latency and energy consumption on compute-bound systems.
• Used to calculate operational intensity for roofline analysis.
• Pruning and efficient architectures directly reduce this count.

Throughput is the sustained rate of inference processing, measured in Frames per Second (FPS) or Inferences per Second (IPS). It measures system capacity for batch or continuous streaming data.

• Critical for processing sensor streams (e.g., video, audio).
• Maximized by pipelining and minimizing idle time between inferences.
• Often trades off against single-inference latency.
• Example: A vibration-based predictive maintenance system may require 100 IPS to analyze high-frequency sensor data.

The accuracy-efficiency trade-off is the fundamental engineering compromise. Improving a model's predictive accuracy (e.g., Top-1 %) typically increases its size, latency, and energy cost. The optimal model lies on the Pareto frontier.

• Quantified by plotting accuracy against metrics like latency, model size, or energy.
• Techniques like quantization and pruning aim to move this frontier, reducing cost with minimal accuracy loss.
• The target operating point is dictated by the application's minimum viable accuracy and hard resource constraints.

Peak memory usage is the maximum RAM/SRAM consumed during inference, encompassing model weights, activations, and intermediate buffers.

• The primary constraint for microcontroller deployment, where RAM is often limited to kilobytes.
• Dictates model feasibility on a given hardware target.
• Reduced via techniques like activation spilling, quantization (reducing weight precision), and model pruning.
• Profiling tools measure layer-by-layer memory allocation to identify bottlenecks.

Energy per inference is the total electrical energy consumed to complete a single forward pass, typically measured in microjoules (µJ).

• The key metric for battery-operated edge devices.
• Calculated as average power (watts) multiplied by inference latency (seconds).
• Composed of dynamic power (from computation/data movement) and static power (leakage).
• Optimization focuses on reducing MACCs, minimizing memory accesses, and managing processor states (e.g., sleep modes).

MACC count (Multiply-Accumulate Operations) is the total number of fused multiplication and addition operations required for a forward pass.

• A fundamental proxy for computational workload and complexity.
• Directly correlates with inference time and energy consumption on compute-bound systems.
• Convolutional and fully-connected layers are MACC-intensive.
• Reduced through architectural choices like depthwise separable convolutions and channel pruning.

The accuracy-latency trade-off describes the fundamental engineering compromise where improving a model's predictive accuracy typically increases its computational complexity and inference latency.

• Adding parameters or making a network deeper usually boosts accuracy but also increases MACCs and memory footprint.
• The goal is to find the optimal operating point on the Pareto frontier, where no metric can be improved without degrading another.
• Techniques like neural architecture search (NAS) automate the search for models that balance this trade-off for a given hardware target.

A system is compute-bound when performance is limited by arithmetic unit speed, and memory-bound when limited by data movement bandwidth.

• Compute-bound kernels have high operational intensity (FLOPs/byte). Their performance is near the processor's peak FLOPs rate.
• Memory-bound kernels have low operational intensity. Time is spent waiting for weights/activations to be fetched from memory.
• The Roofline Model is used to diagnose this bound and guide optimization. TinyML models on microcontrollers are often severely memory-bound.