Load Testing

The fundamental pair of metrics defining system responsiveness and capacity.

• Latency: The time delay for a single request, measured from submission to completion. For language models, this is often broken into Time to First Token (TTFT) and Time per Output Token (TPOT).
• Throughput: The rate of request processing, measured in Queries Per Second (QPS) or Tokens Per Second (TPS). Load testing reveals the throughput-latency curve, showing how latency degrades as throughput increases toward the system's saturation point.

Measures the worst-case experience, critical for user perception and SLOs.

• Percentile Latency (P50/P90/P99): Statistical measures of latency distribution. P50 is the median; P99 represents the slowest 1% of requests.
• Tail Latency: High-percentile latencies (P95, P99+) that often determine overall user-perceived performance. Spikes can indicate resource contention, garbage collection, or uneven workload distribution.
• Load tests must sustain pressure long enough to capture these tail events, which are often missed in short-duration tests.

Measures how efficiently system hardware is used, identifying bottlenecks.

• Hardware Utilization: Percentage of available computational resources (GPU/CPU cores, memory bandwidth) actively engaged.
• Bottleneck Analysis: Determines if the workload is compute-bound (limited by FLOPs) or memory-bound (limited by data movement). Tools like the Roofline Model help visualize this.
• Key metrics include GPU memory usage, CPU usage, and memory bandwidth consumption. High utilization with low throughput indicates inefficiency.

Quantifies reliability under load, ensuring graceful degradation.

• Measures the percentage of requests that fail or time out as load increases. A rising error rate at high load indicates the system's failure mode.
• Tracks stability over extended test durations to identify memory leaks or performance degradation.
• Provides data to define Service Level Indicators (SLIs) like success rate, which underpin Service Level Objectives (SLOs).

Defines the system's capacity limits and scaling behavior.

• Concurrent Requests: The number of requests being processed simultaneously. Tests measure how latency and throughput scale with concurrency.
• Saturation Point: The load level where throughput plateaus and latency increases exponentially. The goal of capacity planning is to operate safely below this point.
• Identifies the maximum supported concurrency before resource contention causes thrashing or failures.

Distinguishes between initial initialization and normal operating performance.

• Cold Start Latency: The delay for the first request(s) after startup, encompassing model loading, kernel compilation, and cache warming.
• Steady-State Performance: The consistent latency and throughput achieved after the system is fully warmed up.
• Load tests should measure both phases, as cold start is critical for auto-scaling environments and serverless deployments.

Latency is the time delay between a request being submitted to an inference system and the corresponding response being received, measured in milliseconds. It is the primary user-facing performance metric. Key latency types include:

• Time to First Token (TTFT): The delay until the first output token is generated, representing initial processing overhead.
• Time per Output Token (TPOT): The average time to generate each subsequent token in a stream.
• Tail Latency (P95/P99): The high-percentile latencies experienced by the slowest requests, critical for user-perceived performance.

Throughput is the rate at which an inference system processes requests, measuring its capacity. It is typically expressed in:

• Queries Per Second (QPS): The number of complete inference requests processed per second.
• Tokens Per Second (TPS): The total number of output tokens generated per second, a key metric for language models. Throughput and latency have an inverse relationship; as load increases toward the system's saturation point, throughput plateaus and latency rises sharply, defining the operational limits.

Stress testing subjects an inference system to loads beyond its expected operational capacity to identify breaking points and failure modes. Unlike load testing, which simulates realistic traffic, stress testing pushes the system to its absolute limits to answer critical questions:

• What is the maximum throughput before failure?
• How does the system degrade (e.g., error rates, latency spikes)?
• What component fails first (e.g., GPU memory, network)? This reveals the system's safety margin and resilience under extreme conditions.

A performance baseline is a set of established metrics and measurements that serve as a reference point for all future performance comparisons. It is captured under controlled, reproducible conditions and typically includes:

• Latency distributions (P50, P90, P99).
• Throughput at target load levels.
• Hardware utilization (GPU, CPU, memory). Baselines are essential for detecting performance regressions after software updates, configuration changes, or model deployments, enabling objective, data-driven evaluation of system changes.

A Service Level Objective (SLO) is a specific, measurable target for system performance or reliability that forms the basis of operational agreements. For inference systems, SLOs are typically defined using Service Level Indicators (SLIs) like latency or error rate. Examples include:

• "P99 latency shall be under 200ms for 99.9% of requests over a 30-day period."
• "System throughput shall not degrade below 1000 QPS." Load testing is the primary method for validating that a system architecture can meet its defined SLOs before deployment.