Inference Latency

The time required to transform raw input data into the format expected by the model. This stage is often a hidden bottleneck on edge devices.

• Key Operations: Resizing images, audio sampling, normalization, tokenization for text.
• Edge Challenge: Limited CPU resources can make preprocessing surprisingly costly, especially for high-resolution sensor data.
• Optimization: Use fixed-point arithmetic, hardware-accelerated libraries (e.g., OpenCV with NEON instructions), and pre-computed lookup tables.

The time to read the model from storage (flash memory) into RAM, decompress it if necessary, and prepare it for execution. This is a one-time cost per session but critical for application startup time.

• Factors: Model file size, storage I/O speed, decompression overhead, runtime initialization (e.g., allocating intermediate buffers).
• Edge Impact: On devices with slow eMMC storage, this can dominate cold-start latency.
• Mitigation: Use model serialization formats optimized for fast loading, keep a decompressed model in volatile memory if possible, and employ just-in-time compilation strategies.

The core computational phase where the model's mathematical operations (layers) are executed on the hardware. This is typically measured in FLOPs (Floating-Point Operations) or MACs (Multiply-Accumulate Operations).

• Primary Driver: The model's architecture, parameter count, and the efficiency of the underlying compute kernels.
• Hardware Dependence: Speed varies drastically between a CPU, GPU, NPU, or DSP. Kernel execution leverages SIMD (Single Instruction, Multiple Data) and specialized instructions.
• Optimization: Use model compression (pruning, quantization), operator fusion, and hardware-specific kernel libraries (e.g., ARM Compute Library, TensorFlow Lite for Microcontrollers).

The latency incurred by moving data between different levels of the memory hierarchy. This is often the dominant factor, not raw compute, due to the von Neumann bottleneck.

• Hierarchy: Weights and activations move from DRAM → cache → registers. Each hop has different latency and bandwidth.
• Bottlenecks: Loading weights for large layers and writing/reading large intermediate activation maps.
• Techniques: Model compression reduces the size of weights and activations. Data layout optimization (NHWC vs. NCHW) improves cache locality. Operator fusion reduces intermediate writes.

The overhead introduced by the software runtime to manage parallel execution, especially when handling multiple inputs (batches) or concurrent models.

• Dynamic Batching: Grouping multiple inference requests to amortize kernel launch overhead and improve hardware utilization.
• Edge Constraint: On resource-constrained devices, large batch sizes increase memory footprint and per-request latency. Continuous batching is a more advanced technique for streaming inputs.
• Scheduling: The runtime must decide the order of layer execution and manage dependencies, which can introduce stalls.

The time to interpret the model's raw output tensor into a usable result. This can be computationally significant.

• Common Tasks: Applying a Softmax function for classification, running Non-Maximum Suppression (NMS) for object detection, decoding sequences for speech-to-text, or filtering and thresholding.
• Optimization: Fuse final layers (e.g., Softmax) into the model graph. Use integer-optimized versions of algorithms like NMS. Offload to a dedicated co-processor if available.

Model throughput measures the number of inferences a system can process per second (e.g., inferences/sec or FPS). While latency is a per-inference delay, throughput is a measure of batch processing capacity.

• Key Relationship: High throughput often requires batching inputs, which can increase latency for the first item in the batch. Optimizing for low latency (real-time response) can reduce maximum throughput.
• Edge Consideration: On resource-constrained devices, the goal is often to maximize throughput within a strict latency Service Level Agreement (SLA), balancing the two metrics based on the application's needs.

First token latency (or time-to-first-token) is a critical metric for autoregressive models, like Large Language Models (LLMs), measuring the delay from receiving the input prompt to generating the very first output token.

• Primary Bottleneck: This initial delay encompasses the full forward pass of the prompt through the model and is heavily influenced by prompt length and prefill computation.
• Distinction from Per-Token Latency: After the first token, generation continues with a faster per-token latency. For streaming applications, low first token latency is essential for perceived responsiveness.

Tail latency refers to the worst-case inference delays, typically measured at high percentiles like the 95th (p95) or 99th (p99). It represents the long tail of the latency distribution.

• Systemic Importance: While average latency might be low, high tail latency causes unpredictable jitter, breaking real-time applications. It is often caused by resource contention, garbage collection, thermal throttling, or interrupts on edge devices.
• Mitigation: Requires deterministic system design, priority scheduling, and robust performance profiling under all operational conditions.

Hardware inference time is the core component of total latency, measuring the time the processor (CPU, GPU, NPU, TPU) takes to execute the model's computational graph. It excludes all system overhead.

• Determinants: Dictated by the model's FLOPs, memory bandwidth requirements, and how well the model's operators are mapped to the hardware's parallel execution units.
• Optimization Levers: Reduced via model compression (pruning, quantization), kernel optimization, and using hardware-specific AI compilers (e.g., TensorRT, OpenVINO, XNNPACK) that generate highly efficient code for the target silicon.

End-to-End (E2E) latency is the total observed delay from the arrival of a raw input signal (e.g., a camera frame) to the delivery of a usable final result. It is the metric that matters to the end-user.

• Comprehensive Scope: Includes pre-processing (e.g., image resizing, audio framing), hardware inference time, post-processing (e.g., non-maximum suppression, decoding), and any system I/O or communication overhead.
• Edge Bottlenecks: On edge devices, sensor data acquisition and memory-bound pre/post-processing can often dominate latency more than the model inference itself.

Real-time inference describes a system where the E2E latency is guaranteed to be below a strict, application-defined deadline for every input, ensuring correct and timely operation.

• Hard vs. Soft Real-Time: Hard real-time systems (e.g., autonomous braking) fail catastrophically if deadlines are missed. Soft real-time systems (e.g., video call background blur) degrade gracefully.
• Achievement Requirements: Requires deterministic performance at the 99.9th percentile (p999) latency, achieved through a combination of optimized models, real-time operating systems (RTOS), and hardware with predictable execution pipelines.