Latency Under Load

The core software stack for executing model computations. Continuous batching dynamically groups incoming requests to maximize GPU utilization, but poor implementation leads to head-of-line blocking.

• Static vs. Dynamic Batching: Static batching waits to fill a fixed batch size, increasing tail latency. Dynamic batching groups requests as they arrive.
• Iteration-Level Scheduling: Advanced schedulers like ORCA or vLLM's PagedAttention allow partial execution of sequences, improving throughput but adding scheduling overhead.
• Kernel Fusion: Optimized GPU kernels that combine operations reduce data transfer between GPU memory and cores, a critical factor at high load.

Physical compute and memory limits under concurrent execution. The GPU memory bus and VRAM capacity are primary bottlenecks.

• KV Cache Pressure: The Key-Value cache for autoregressive generation grows with batch size and sequence length. Exhausting VRAM forces slow swaps to system RAM.
• Memory-Bound vs. Compute-Bound: Many transformer operations are memory-bound, meaning latency is dictated by the speed of loading weights from VRAM, not by FLOPs.
• NUMA Effects: In multi-socket servers, non-uniform memory access can drastically increase latency if processes are not pinned to local CPU and memory nodes.

The strategy for handling long prompts within the model's fixed context limit. Attention computation scales quadratically with sequence length, making it a major latency driver.

• Sliding Window Attention: Limits the context each token can attend to, reducing compute but potentially losing distant information.
• Context Caching & Eviction: Systems that cache computed context for repeated prompts (e.g., in chat applications) must implement eviction policies under memory pressure.
• Prompt/Output Token Imbalance: Processing a 10k-token input is far more costly than generating a 100-token output, causing uneven load.

Latency introduced by the surrounding service infrastructure before and after the core model inference.

• Load Balancer Queuing: Incoming requests queue at the load balancer or API gateway. Under overload, queueing delay dominates total latency.
• Model Sharding & Pipeline Parallelism: Distributing a large model across multiple GPUs reduces per-device memory load but introduces communication latency between devices.
• Cold Starts & Autoscaling Lag: Serverless or containerized deployments experience significant latency spikes when scaling from zero or adding new replicas.

Inherent characteristics of the model that determine its computational footprint. Smaller models and architectural optimizations directly reduce latency.

• Quantization: Using lower precision (e.g., FP16, INT8, INT4) reduces memory footprint and increases compute speed but may affect output quality.
• Sparsity & Pruning: Removing insignificant weights (pruning) or leveraging inherently sparse architectures (Mixture of Experts) reduces FLOPs.
• Operator Optimization: Use of fused operators (like FlashAttention-2) and hardware-specific kernels (for NVIDIA Tensor Cores) maximizes hardware efficiency.

The ability to measure and diagnose the root cause of latency spikes. Without granular metrics, optimization is guesswork.

• Per-Layer Profiling: Tools like PyTorch Profiler or NVIDIA Nsight Systems identify if latency is in attention layers, feed-forward networks, or embedding lookups.
• Tail Latency Metrics (P99, P99.9): Average latency hides outliers. High percentiles reveal blocking issues affecting a subset of requests.
• GPU Utilization vs. SMs Active: High overall GPU utilization can mask starvation, where streaming multiprocessors (SMs) are idle waiting for memory loads.

The number of inference requests a system can process per unit of time (e.g., requests per second). While latency measures the time for a single request, throughput measures overall system capacity. Under load, these metrics trade off: increasing throughput often increases average latency due to queuing and resource contention. Continuous batching is a key optimization technique that groups requests to maximize throughput without proportionally increasing latency.

The high-percentile response times (e.g., p95, p99) experienced by the slowest requests under load. While average latency gives a general view, tail latency is critical for user experience, as it defines the worst-case delays. It is heavily influenced by system variance, garbage collection pauses, and resource saturation. Reducing tail latency often requires optimizing memory access patterns, implementing efficient scheduling, and provisioning headroom.

A performance testing methodology that simulates expected or peak user traffic on a system to measure its behavior under stress. For AI inference, this involves:

• Generating synthetic or replaying real request patterns.
• Gradually increasing the requests per second (RPS) to find breaking points.
• Monitoring key metrics: latency, throughput, error rates, and compute utilization (GPU/CPU). The goal is to identify bottlenecks before deployment and establish service level objectives (SLOs).

The mathematical study of waiting lines, or queues. It provides models (e.g., M/M/1, M/G/1) to predict how arrival rate and service rate affect average wait time (latency) and queue length. Key concepts include:

• Little's Law: The average number of requests in a system equals the arrival rate multiplied by the average time in the system.
• Utilization: As the arrival rate approaches the service rate, latency increases non-linearly. This theory is foundational for capacity planning and understanding latency under load.

The cloud infrastructure capability to automatically add or remove compute resources (e.g., inference endpoints) based on real-time demand. It directly impacts latency under load by:

• Scale-out: Adding replicas to handle increased traffic, distributing load to maintain low latency.
• Scale-in: Removing idle replicas to control cost. Challenges include cold-start latency when new instances spin up and configuring metrics (e.g., CPU utilization, request queue depth) to trigger scaling proactively.

The maximum number of inference requests a system or model instance can process simultaneously. This is a hard constraint often set by:

• Hardware limits: GPU memory capacity for model weights and KV caches.
• Software configuration: Web server workers or framework settings. When the number of concurrent requests exceeds this limit, new requests are queued, directly increasing latency. Optimizing this limit involves profiling memory usage and implementing efficient context switching.