End-to-End Latency

This initial stage captures the raw physical signal (e.g., audio, vibration, image) and prepares it for the model. Latency here is determined by:

• Sensor sampling rate and wake-up time from sleep.
• Analog-to-digital converter (ADC) conversion time.
• Preprocessing steps like filtering, normalization, or windowing applied to the raw signal. For example, a 16kHz audio sample with a 30ms window inherently adds 30ms of acquisition latency before any computation begins.

Before inference, raw sensor data is often transformed into a feature vector suitable for the model. This stage can be a major bottleneck.

• Classical algorithms like Fast Fourier Transform (FFT) for audio or edge detection for vision add computational overhead.
• On-device feature extraction avoids cloud transmission but consumes MCU cycles.
• Optimized kernels (e.g., CMSIS-DSP for Arm Cortex-M) are critical to minimize latency in this stage, which is often memory-bound due to data movement.

The core inference latency of the neural network's forward pass. Key factors include:

• Model architecture complexity (MACC count, layer depth).
• Hardware acceleration via a Neural Processing Unit (NPU) or DSP versus a standard CPU core.
• Memory hierarchy efficiency; frequent access to slow flash memory for weights increases latency.
• Kernel optimization using specialized libraries like TensorFlow Lite for Microcontrollers or CMSIS-NN. This stage is typically profiled via layer-wise profiling to identify bottlenecks.

After the model outputs a tensor (e.g., class probabilities), application logic interprets the result. Latency contributors are:

• Decoding operations like applying a Connectionist Temporal Classification (CTC) decoder for speech-to-text.
• Thresholding and smoothing filters (e.g., to debounce a keyword detection).
• State machine logic that may require historical context from previous inferences. While often lightweight, poorly optimized post-processing can negate gains from efficient inference.

The latency introduced by the embedded operating system and system architecture.

• Real-time OS (RTOS) scheduling delays and context-switching overhead.
• Inter-process communication if tasks are split across cores or processes.
• Peripheral bus contention (I2C, SPI) if communicating with external components.
• Interrupt service routine (ISR) handling time. Achieving deterministic execution requires minimizing and bounding this overhead, which is measured via Worst-Case Execution Time (WCET) analysis.

The final stage where the system acts on the decision. Latency is governed by:

• Actuator response time (e.g., a servo motor, relay, or display update).
• Communication protocol latency if the output is a wireless transmission (BLE, LoRaWAN).
• Power management delays, such as waking a high-power radio from a deep sleep state. In a closed-loop control system, this stage's latency directly impacts system stability and must be included in the end-to-end analysis.

The choice of neural network topology is the primary determinant of computational load. Model pruning—the systematic removal of redundant weights, neurons, or channels—reduces the MACC count and memory footprint, directly lowering inference time. For example, pruning a convolutional layer by 50% can reduce its execution time proportionally. Techniques like magnitude-based pruning and structured pruning are essential for creating lean architectures suitable for microcontrollers, moving the system away from being compute-bound.

Quantization converts model parameters and activations from 32-bit floating-point to lower-precision formats (e.g., 8-bit integers). This drastically reduces:

• Model size (4x reduction for INT8 vs. FP32).
• Memory bandwidth requirements.
• Energy consumption per operation. Microcontrollers excel at fixed-point arithmetic, making quantized models significantly faster. Post-training quantization (PTQ) and quantization-aware training (QAT) are key methods. This optimization directly targets systems that are memory-bound, as it reduces the data movement between SRAM and the compute unit.

Generic neural network operators are inefficient on constrained hardware. Kernel optimization involves hand-tuning or auto-generating low-level code (often in C or assembly) for critical operations like convolutions or matrix multiplies to exploit specific microcontroller features:

• Single Instruction, Multiple Data (SIMD) units.
• Specialized DSP instructions.
• Memory hierarchy (register/cache usage). Frameworks like TensorFlow Lite for Microcontrollers and CMSIS-NN provide optimized kernels for Arm Cortex-M processors. This work ensures the compute unit operates near its theoretical peak, as visualized in a roofline model.

A major source of latency is sequential execution: wait for sensor data, then process, then output. Pipelining breaks the workflow into stages (e.g., data acquisition, preprocessing, inference, postprocessing) that can overlap. While one inference runs, the next sensor sample can be readied in a buffer. This requires double-buffering and careful real-time scheduling to hide I/O latency behind computation, improving overall throughput and reducing the perceived end-to-end latency. This is critical for achieving deterministic execution in real-time systems.

The latency journey begins at the sensor. Optimizations include:

• Reducing sampling rate to the minimum required by the application.
• Using hardware accelerators (e.g., ADC, DMA) for data collection to offload the CPU.
• Simplifying preprocessing steps (e.g., downsampling, filtering, feature extraction). Implementing a filter in the time domain instead of the frequency domain can save milliseconds. The goal is to minimize the data payload and preparation time before it enters the model, addressing bottlenecks that occur long before inference.

Microcontrollers have a limited, tiered memory system (registers, SRAM, flash). Peak memory usage dictates the minimum SRAM required. Optimization strategies:

• Layer fusion: Combining consecutive operations (e.g., batch normalization + activation) to avoid writing intermediate tensors back to slow memory.
• In-place computation: Overwriting input buffers with outputs where possible.
• Strategic tiling of large operations to fit within cache/SRAM. Poor management leads to constant cache thrashing and makes the system memory-bound. Tools that provide layer-wise profiling of memory access are essential for this analysis.