Prediction Latency

Total prediction latency is the sum of several distinct stages:

• Network Transit: Time for the request payload to travel from client to server and the response to return.
• Queueing Delay: Time the request spends waiting in a buffer while the server processes other requests.
• Pre-processing: Time to deserialize, validate, and transform input features into the tensor format the model expects.
• Compute Inference: The actual forward pass of the neural network, including matrix multiplications and attention mechanisms.
• Post-processing: Time to decode output logits, apply business logic, and serialize the prediction into a response payload.

Optimizing any single stage in isolation rarely solves systemic latency problems.

Average latency masks the worst-case experience. P99 latency—the value below which 99% of requests complete—is the critical metric for user-facing systems.

• Long-tail problem: A small fraction of requests can take 10-100x longer than the median due to garbage collection pauses, resource contention, or batch padding.
• Amplification effect: In microservice architectures, a single user request may fan out to dozens of internal inference calls. The probability of hitting a tail latency event compounds multiplicatively.
• Mitigation strategies: Hedged requests (sending the same request to multiple replicas and using the first response) and request cancellation upon timeout.

Cold start refers to the additional latency incurred when a model instance serves its first request after initialization or a period of inactivity.

• Weight loading: Large models must read billions of parameters from disk or object storage into GPU or CPU memory before inference can begin.
• Just-in-time compilation: Frameworks like TensorRT or XLA may compile optimized kernels on first execution, adding seconds of one-time overhead.
• Connection warm-up: Database connection pools, feature store clients, and gRPC channels must be established.

Mitigation: Keep model replicas warm with periodic health-check pings, use lazy loading with pre-fetching, and pre-compile optimized execution graphs during deployment.

Batching groups multiple inference requests to amortize overhead and maximize hardware utilization, but introduces a fundamental tension:

• Static batching: Waits for a fixed batch size or time window to fill before processing. Increases throughput but adds queueing latency proportional to the wait time.
• Dynamic batching: The server continuously aggregates arriving requests and processes them as soon as the compute kernel is free. Reduces idle time but still introduces variable delay.
• Continuous batching: For generative models, evicts completed sequences mid-batch and inserts new requests immediately, eliminating the straggler problem where the batch waits for the longest sequence.

Key insight: The optimal batch size minimizes the sum of compute time and queueing delay for a given latency SLO.

Specialized hardware dramatically reduces compute latency compared to general-purpose CPUs:

• GPU acceleration: Massively parallel architectures execute matrix multiplications in a fraction of the time. NVIDIA Tensor Cores accelerate mixed-precision operations.
• Kernel fusion: Combining multiple operations into a single GPU kernel eliminates intermediate memory reads and writes. Frameworks like TensorRT perform aggressive graph-level fusion.
• FlashAttention: An IO-aware exact attention algorithm that tiles the attention matrix computation to minimize data movement between GPU HBM and on-chip SRAM, reducing latency by 2-4x for long sequences.
• NUMA affinity: Pinning inference processes to specific CPU sockets and their local memory banks avoids cross-socket memory access penalties that can add microseconds per request.

Reducing the computational footprint of the model itself directly lowers inference latency:

• Quantization: Converting 32-bit floating-point weights to INT8 or FP8 reduces memory bandwidth requirements and enables faster integer math. Post-training quantization can achieve 2-4x speedups with minimal accuracy loss.
• Knowledge distillation: Training a compact student model to mimic a larger teacher model preserves predictive quality while reducing parameter count and FLOPs.
• Pruning: Removing weights with near-zero magnitudes or entire attention heads creates sparse models that skip unnecessary computation.
• Speculative decoding: A small draft model predicts multiple future tokens inexpensively, which a large target model then verifies in parallel. This can accelerate autoregressive generation by 2-3x without changing the output distribution.