Steady-State Performance

In steady-state, the latency for processing requests follows a predictable statistical distribution. Key metrics stabilize:

• P50 (Median) Latency: Represents the typical user experience.
• P90/P99 (Tail) Latency: Represents the worst-case delays for a small percentage of requests. A hallmark of steady-state is that these high-percentile latencies are bounded and predictable, not exhibiting wild spikes caused by cold starts or cache misses.
• Time per Output Token (TPOT): For autoregressive models like LLMs, the incremental time to generate each token becomes consistent.

This is the maximum Queries Per Second (QPS) or Tokens Per Second (TPS) the system can handle while maintaining acceptable latency bounds. It is not a peak burst rate but a sustained rate.

• Defined by the Saturation Point: The load level where adding more requests causes latency to increase non-linearly.
• Throughput-Latency Curve: In steady-state, operating points on this curve are stable. Engineers select a target throughput (e.g., 80% of max) to leave headroom for traffic spikes and avoid the 'cliff' where latency degrades rapidly.

System resources operate at efficient, consistent levels:

• GPU/CPU Utilization: High and stable, indicating compute resources are fully engaged but not over-saturated.
• Memory Usage: Constant, with KV Caches for transformer models fully populated and managed. There is no overhead from frequent model loading or cache warming.
• Memory Bandwidth & Compute Balance: The workload's behavior aligns with the system's capabilities, as analyzed by tools like the Roofline Model. The system is neither severely compute-bound nor memory-bound in an unpredictable way.

Steady-state is defined by the completion of one-time or periodic startup costs:

• No Cold Start Latency: The model is loaded, compiled (e.g., via TensorRT or TorchInductor), and resident in GPU memory.
• Warm Caches: Critical data structures like the KV Cache for attention mechanisms are pre-allocated and warmed. For retrieval-augmented generation (RAG) systems, vector index embeddings are loaded.
• JIT Compilation Complete: Just-in-time compilation kernels for the specific hardware are cached and ready.

A system in steady-state responds predictably to changes in concurrency and request patterns.

• Managed Concurrent Requests: The system efficiently schedules multiple requests using techniques like continuous batching, without causing severe resource contention.
• Graceful Degradation: If load exceeds the sustainable throughput, latency increases predictably (e.g., following a queueing theory model like M/M/1), rather than failing catastrophically.
• Effective Load Balancing: In distributed setups, traffic is evenly spread across replicas, preventing hot spots.

The initial delay before an inference system reaches steady-state performance. This includes overhead from model loading, compilation, cache population, and resource allocation. Key phases:

• Model Loading: Reading weights from disk into GPU memory.
• Kernel Compilation: Just-in-time compilation of operators for the specific hardware.
• Cache Warming: Populating the KV Cache with initial computations.

This latency is excluded from steady-state measurements, which begin only after the system is fully warmed up.

A fundamental graph showing the relationship between a system's request throughput and its corresponding latency as load increases. It is essential for characterizing steady-state behavior.

Typical curve progression:

• Low Load: Latency is stable and minimal.
• Knee Point: Latency begins to increase non-linearly as throughput approaches system capacity.
• Saturation: Throughput plateaus at its maximum while latency increases sharply due to queuing.

Engineers use this curve to identify the optimal operating point that balances high throughput with acceptable latency for their Service Level Objectives (SLOs).

The high-percentile latencies (e.g., P95, P99) that represent the slowest requests in a distribution. While average latency might be good, tail latency determines user-perceived performance.

Causes in steady-state:

• Resource Contention: Spikes when multiple requests compete for GPU memory or compute.
• Garbage Collection: Pauses in the runtime environment.
• Noisy Neighbors: Interference from other processes on shared hardware.

Steady-state performance analysis requires monitoring tail latency, not just averages, to ensure consistent quality of service.

The maximum sustainable throughput of an inference system under steady-state conditions. Beyond this point, the system cannot process requests faster, leading to:

• Exponential growth in request queueing.
• Violations of latency SLOs.
• Increased error rates.

Identifying Saturation: It is found at the 'knee' of the throughput-latency curve. Operating near but below the saturation point is the goal for cost-efficient, high-utilization deployment. Exceeding it leads to performance collapse.

A set of established steady-state performance metrics (latency, throughput, resource utilization) recorded under a controlled, reproducible workload and environment. This serves as the reference for all future comparisons.

Uses:

• Detecting Regressions: Quantifying the impact of a new model version, library update, or configuration change.
• Capacity Planning: Forecasting hardware needs based on traffic growth.
• SLO Definition: Providing data-driven targets for latency and throughput.

A valid baseline must be measured after warm-up, under defined load, and with documented hardware and software configurations.

The competition between concurrent inference requests (or other processes) for finite system resources, which is a primary source of instability during steady-state operation.

Common contention points:

• GPU Memory Bandwidth: Multiple requests reading/writing activations and KV caches.
• Compute Cores (SM): Warps from different requests waiting for execution units.
• Host CPU: Pre/post-processing bottlenecks.
• PCIe Bus: Data transfer between CPU and GPU.

Techniques like continuous batching and effective scheduling are designed to minimize contention and maintain predictable steady-state performance.