Inference Latency

This initial stage captures the time to read raw sensor data (e.g., from an accelerometer or microphone) and prepare it for the model. Latency here is governed by:

• Sensor sampling rate and communication bus speed (I²C, SPI).
• Preprocessing algorithms like digital filtering, Fast Fourier Transforms (FFT), or normalization.
• Buffering requirements to accumulate enough samples for a single inference window. For a 16kHz audio keyword spotting model, acquiring a 1-second window inherently adds 1,000 ms of latency before computation even begins.

The core computational phase where the neural network processes the input tensor. Latency is determined by:

• Model complexity: Number of parameters, layers, and operations (MACs).
• Hardware compute: Clock speed of the MCU core and presence of accelerators like the Arm Ethos-U55 microNPU or DSP blocks.
• Software efficiency: Quality of the inference runtime (e.g., TensorFlow Lite Micro, CMSIS-NN kernels) and use of optimizations like operator fusion. This is often the primary focus of optimization via model quantization, pruning, and hardware-aware NAS.

The movement of model weights and intermediate activations (tensors) between memory types creates significant latency. Key factors include:

• Flash vs. RAM Speed: Reading weights from Flash memory is orders of magnitude slower than from SRAM. Techniques like layer-by-layer fetching minimize RAM footprint but can increase latency.
• Memory Bandwidth: The data bus width (32-bit vs. 64-bit) limits how quickly data can be moved to the compute unit.
• Cache Effects: Small CPU caches on MCUs are critical. Poor locality of memory accesses leads to frequent cache misses and stalls. Optimizing the memory footprint and access pattern is as important as reducing compute operations.

In real-time embedded systems, the inference task does not own the CPU. Latency spikes occur due to:

• High-priority interrupts from other system components (e.g., radio communication, timers).
• Operating system overhead from task scheduling, if an RTOS is used.
• Garbage collection in managed runtimes, which can cause non-deterministic pauses. For deterministic latency, TinyML systems often use bare-metal programming or assign inference to a high-priority RTOS task, and employ static memory allocation to avoid runtime heap management.

The time after inference to interpret the model's output and trigger an action. This includes:

• Applying a softmax or sigmoid function to raw logits.
• Running decision algorithms (e.g., smoothing filters, thresholding, state machines) to reduce false positives. A keyword spotter may require a trigger word to be detected in 2 out of 3 consecutive windows.
• Actuation delay to control an output (e.g., turning on an LED, sending a message via UART). While often lightweight, poorly designed post-processing can add non-trivial, variable delay.

Latency is intrinsically linked to power consumption, governed by the device's dynamic voltage and frequency scaling (DVFS).

• Higher clock speeds reduce compute latency but increase power draw quadratically.
• Low-power modes: Devices often sleep between inferences. The wake-up and clock stabilization time adds to the overall periodic latency.
• Metric: Inferences Per Joule (IPJ) encapsulates this trade-off. Optimizing for low latency can drastically reduce battery life. Techniques like model cascades use a tiny, fast 'wake-up' model to gate the execution of a larger, more accurate model, optimizing the overall system efficiency.

Worst-Case Execution Time (WCET) is the maximum possible time a specific computational task, such as a single model inference pass, could take to complete on an edge device under all possible inputs and system states. For deterministic real-time TinyML systems (e.g., in automotive or industrial control), guaranteeing a WCET is more critical than average latency. Analysis involves:

• Accounting for all possible code paths and data-dependent branches.
• Modeling memory hierarchy effects (cache hits/misses).
• Considering interference from other system tasks or interrupts. A model must be designed and compiled such that its WCET meets the hard deadline of the application.

Milliwatt computing refers to the design and execution of software, particularly machine learning inference, on hardware systems that operate within a total power budget of milliwatts (mW). This ultra-low-power domain is the target for battery-powered or energy-harvesting TinyML devices. Key implications for latency include:

• A direct power-performance trade-off: higher clock speeds reduce latency but increase dynamic power consumption quadratically.
• Use of power gating and dynamic voltage and frequency scaling (DVFS) to idle unused components, which can add wake-up latency.
• The primary metric becomes Inferences Per Joule (IPJ), balancing latency against total energy consumed per prediction.

These are the two fundamental throughput and efficiency metrics that contextualize inference latency.

• Inferences Per Second (IPS): The reciprocal of average latency (1/latency). Measures raw throughput when the system is continuously processing inputs. For a model with 20 ms latency, maximum IPS is 50.
• Inferences Per Joule (IPJ): The number of inferences a device can perform per joule of energy consumed. This is the key energy-efficiency metric for battery life. It incorporates both the latency and the power draw during inference: IPJ = IPS / Power (Watts). Optimizing for low latency (high IPS) without regard for power can drastically reduce IPJ and total device runtime.

Static memory allocation is a memory management strategy where all RAM required for a program's execution—including buffers for machine learning model weights, activations, and intermediate tensors—is allocated at compile-time. This is critical for deterministic latency in TinyML because it:

• Eliminates runtime overhead of dynamic allocation (malloc/free), which is unpredictable and can cause fragmentation.
• Allows the compiler to precisely map the entire memory footprint, enabling aggressive optimizations and guaranteeing the model will run if it fits.
• Ensures the worst-case memory usage is known and bounded, which is a prerequisite for calculating a reliable WCET. Frameworks like TensorFlow Lite for Microcontrollers use static allocation for this reason.

Hardware-Aware Neural Architecture Search (HW-NAS) is an automated process for designing neural network architectures where the search algorithm directly optimizes for hardware performance metrics like latency, energy consumption, and memory usage on the target device. Unlike standard NAS that only optimizes for accuracy, HW-NAS treats the hardware as a first-class constraint. It works by:

• Using a latency/energy lookup table or a predictor model that estimates the cost of each neural network operation (op) on the target MCU or NPU.
• Incorporating these cost estimates into the search algorithm's objective function (e.g., Accuracy / (Latency * Memory)).
• Producing models that are not just accurate, but are Pareto-optimal for the specific latency and resource constraints of the deployment hardware, such as an Arm Cortex-M4 or Ethos-U55.

Operator fusion is a compiler-level optimization that combines multiple sequential neural network operations (layers) into a single, compound kernel. This is a major technique for reducing inference latency on edge devices. For example, a Convolution → Batch Normalization → Activation sequence can be fused into one op. Benefits include:

• Reduced Memory Accesses: Intermediate results (tensors) are kept in registers or fast cache instead of being written to and read from slow main memory (SRAM).
• Elimination of Kernel Launch Overhead: Executing one fused kernel instead of three separate ones reduces scheduling and function call overhead.
• Improved Data Locality: Fused loops can perform more computation per data fetch. Compilers like Apache TVM, TensorFlow Lite, and specialized MCU inference engines (e.g., TinyEngine in MCUNet) perform aggressive operator fusion to minimize latency.