Performance Baseline

Latency measures the time delay for request processing. A comprehensive baseline includes:

• Time to First Token (TTFT): The initial delay before token generation begins, dominated by prompt processing.
• Time per Output Token (TPOT): The average time to generate each subsequent token in an autoregressive stream.
• Percentile Latency (P50/P90/P99): Statistical measures of the latency distribution, where P99 (tail latency) is critical for user experience.
• End-to-End Latency: The total time from client request to final response receipt, including network overhead.

Throughput quantifies the system's processing rate under load. Key metrics are:

• Queries Per Second (QPS): The number of complete inference requests processed per second.
• Tokens Per Second (TPS): The total output tokens generated per second across all concurrent requests.
• Concurrent Request Capacity: The maximum number of simultaneous requests the system can handle while meeting latency Service Level Objectives (SLOs). Throughput is measured across various load levels to create a throughput-latency curve, identifying the system's saturation point.

This component measures hardware efficiency during inference, critical for cost analysis. It includes:

• GPU Utilization: Percentage of streaming multiprocessors (SMs) actively computing.
• GPU Memory Usage: Peak and steady-state consumption of VRAM.
• Memory Bandwidth Utilization: Critical for identifying memory-bound workloads.
• CPU Utilization: Host processor usage, often a bottleneck for pre/post-processing.
• Power Draw: Watts consumed, directly tied to operational expense. Tools like NVIDIA Nsight Systems or AMD ROCprofiler collect this performance telemetry.

SLIs are the measurable metrics that define service quality. For an inference baseline, core SLIs are:

• Latency SLI: e.g., P99 end-to-end latency.
• Throughput SLI: e.g., sustained QPS.
• Availability SLI: e.g., successful request rate (success / total).

Service Level Objectives (SLOs) are the target values for these SLIs (e.g., "P99 latency < 200ms for 99.9% of requests"). The baseline documents the system's performance against these SLOs under defined load.

A baseline is meaningless without a precise definition of the input workload. This includes:

• Request Profile: Input prompt length distribution (mean, max).
• Output Profile: Expected output token length distribution.
• Request Arrival Pattern: Constant request rate, Poisson-distributed (random), or bursty.
• Synthetic vs. Real-World: Whether the load is generated artificially (synthetic workload) or sampled from production logs (real-world workload). This characterization ensures the baseline is reproducible for load testing and performance regression detection.

Every baseline must document the exact hardware and software stack to ensure reproducibility:

• Hardware Specs: GPU model (e.g., NVIDIA H100), CPU, memory, network interconnect.
• Software Versions: OS, driver, CUDA/cuDNN, Triton Inference Server, PyTorch/TensorFlow.
• Model Configuration: Precision (FP16, INT8), graph optimizations, continuous batching window size, KV cache parameters.
• System Settings: Power limits, clock speeds, and any BIOS/firmware tunings. This record is essential for bottleneck analysis and fair comparison after any change.

A Service Level Objective (SLO) is a specific, measurable target for system reliability or performance that forms the basis of a Service Level Agreement. For inference systems, SLOs are typically defined using metrics derived from the performance baseline, such as:

• Maintaining P99 latency below 200ms for 99.9% of requests.
• Achieving a minimum throughput of 1000 queries per second (QPS).
• Ensuring 99.95% service availability over a month. SLOs translate the technical performance baseline into business-critical guarantees.

Performance regression is an undesirable degradation in system performance metrics compared to a previously established baseline. It is detected by continuous comparison against the baseline and can be caused by:

• Software updates or new model versions.
• Configuration changes (e.g., batch size, quantization).
• Changes in underlying infrastructure or resource allocation.
• Increased load or altered request patterns. Identifying regression triggers a root-cause analysis to restore system performance to its expected baseline levels.

Tail latency refers to the high-percentile latencies in a request distribution, representing the slowest requests. It is a critical component of a performance baseline because it determines user-perceived performance.

• P90 Latency: The latency at the 90th percentile; 90% of requests are faster than this value.
• P99 Latency: The latency at the 99th percentile; 99% of requests are faster, representing the "worst-case" for most users. A robust baseline must account for tail latency, not just average latency, to ensure consistent quality of service.

Steady-state performance refers to the consistent latency and throughput characteristics of an inference system after initial transient phases. It is the primary regime for establishing a reliable performance baseline. Key phases include:

1. Cold Start: Initial model loading, compilation, and warm-up.
2. Warm-up: Period where caches (e.g., KV Cache) are populated and execution paths are optimized.
3. Steady-State: Consistent performance where metrics stabilize. Baselines are measured during steady-state to ensure reproducibility and fair comparisons.

The choice of workload is fundamental to defining a meaningful performance baseline.

• Synthetic Workload: An artificially generated set of requests designed for controlled testing. It allows for reproducible, isolated measurement of specific system aspects (e.g., fixed prompt length, uniform distribution).
• Real-World Workload: Consists of actual production traffic, capturing genuine user behavior, variable request patterns, and mixed query complexity. It provides the most accurate baseline for capacity planning. A comprehensive benchmarking strategy uses synthetic workloads for initial profiling and real-world workloads for validation.

A throughput-latency curve is a graphical representation of the fundamental trade-off between system throughput and request latency as load increases. It is a core analytical tool derived from baseline measurements.

• At low load, latency is stable and minimal.
• As load increases, latency begins to rise gradually.
• At the saturation point, throughput peaks and latency increases dramatically due to queueing. This curve, generated from baseline tests, is essential for determining optimal operating points and understanding system limits under the Inference Performance Benchmarking pillar.