Inference Latency

This is the core computational phase where the neural network processes the input tensor. It is dominated by the MACC count and the efficiency of the underlying compute kernels (e.g., CMSIS-NN for Arm Cortex-M). Factors include:

• Layer-wise profiling to identify bottlenecks (e.g., a large dense layer).
• Whether the system is compute-bound (limited by ALU throughput) or memory-bound (limited by weight/activation fetch).
• The impact of model compression techniques like quantization, which reduces data precision to speed up arithmetic.

The time spent moving model weights, activations, and intermediate buffers between memory hierarchies. This is often the dominant latency source in memory-bound systems. Key aspects are:

• Peak memory usage determines the required SRAM, with external flash access being orders of magnitude slower.
• Operational intensity measures compute per byte moved; low intensity indicates a memory bottleneck.
• Techniques like weight caching and activation tiling aim to keep frequently accessed data in faster, on-chip memory to minimize costly off-chip accesses.

The overhead for preparing input data for the model and interpreting its output. This is frequently overlooked but critical for end-to-end latency. Examples include:

• Sensor data processing: Normalizing audio samples, resizing images, or applying digital signal processing filters.
• Output decoding: Converting a probability vector into a class label or parsing bounding boxes for object detection.
• On microcontrollers, these steps may involve fixed-point arithmetic or lookup tables to avoid floating-point operations.

Latency introduced by the operating system, runtime, and interaction with other system tasks. For real-time TinyML deployment, this must be predictable.

• Context switches between the inference task and other firmware routines.
• Interrupt Service Routine (ISR) latency if inference is triggered by a sensor interrupt.
• Deterministic execution is a key goal, ensuring consistent timing devoid of garbage collection pauses or non-deterministic caches.
• Worst-case execution time analysis must account for this overhead.

Characteristics intrinsic to the microcontroller or accelerator silicon that directly constrain achievable latency.

• Clock speed and microarchitecture (e.g., Cortex-M4 vs. M7).
• Presence of a Neural Processing Unit or DSP extensions for accelerated MAC operations.
• Memory hierarchy speeds: SRAM vs. Flash access times.
• Thermal throttling can reduce clock speed on sustained workloads, increasing latency.
• Static power vs. dynamic power trade-offs influence voltage/frequency scaling decisions.

Accurately decomposing latency requires specialized tools and methodologies.

• Profiling tools like Arm Keil MDK Profiler or Segger SystemView trace execution cycles.
• Hardware-in-the-loop testing is essential for valid measurements on the actual target device.
• Performance counters can be used to count cache misses or stalled cycles.
• Statistical profiling or instrumented layer-wise profiling helps attribute time to specific network layers or functions.

The maximum amount of RAM or SRAM consumed during an inference task. This includes the model's weights, intermediate activation buffers, and the runtime's working memory. In TinyML, exceeding the device's available SRAM (often < 512KB) causes catastrophic failure. Profiling must account for the worst-case activation memory across all network layers, not just the model's static size.

The total electrical energy, measured in microjoules (µJ) or millijoules (mJ), required to complete a single forward pass. This is the integral of power over the inference latency. It is the critical metric for battery-powered devices. Key factors include:

• Dynamic Power from compute and memory access.
• Static Power from leakage current.
• The energy cost of waking peripherals (e.g., an ADC or radio).

The sustained rate of inference processing, measured in inferences per second (IPS) or frames per second (FPS). While latency measures a single task's delay, throughput measures system capacity under continuous load. For a pipelined system (e.g., continuous audio classification), high throughput is more critical than ultra-low single-inference latency. Throughput is inversely related to latency only when the system processes inferences serially.

The maximum possible time an inference could take under all permissible operating conditions. Essential for hard real-time systems (e.g., motor control, safety sensors). Determining WCET requires analyzing all possible code paths, accounting for variable loop iterations (e.g., in variable-length input processing), and considering hardware non-determinism like cache behavior. It is always greater than or equal to the measured average latency.

The total delay from the initial sensor data acquisition to the final actionable output. This encompasses stages often omitted from pure model inference measurements:

• Sensor sampling and signal conditioning.
• Data preprocessing (filtering, windowing, feature extraction).
• Inference latency (the model forward pass).
• Post-processing and decision logic.
• Actuation time (e.g., triggering a relay or sending a message).

The high-percentile latencies (e.g., 95th or 99th percentile) in a distribution. While average latency is important, tail latency determines system responsiveness for the worst-case requests. In TinyML, tail latency spikes can be caused by:

• Garbage collection or memory defragmentation events.
• Interrupt handling from other system processes.
• Non-deterministic hardware behavior (e.g., cache misses, DRAM refresh). Managing tail latency is key for predictable user experience.