Stress Testing

The core objective is to determine the absolute maximum capacity of the system before catastrophic failure. This involves incrementally increasing load—measured in Queries Per Second (QPS) or Tokens Per Second (TPS)—until the system experiences:

• Service degradation: Latency exceeds Service Level Objectives (SLOs).
• Resource exhaustion: GPU memory errors, CPU saturation, or out-of-memory (OOM) crashes.
• Complete failure: The service becomes unresponsive or crashes. The result is a clear saturation point and an understanding of the failure mode (e.g., memory-bound vs. compute-bound).

Stress testing evaluates how the system behaves under extreme duress. The goal is not just to find the breaking point, but to ensure the system fails safely and recovers. Key checks include:

• Error handling: Are users presented with clear, actionable error messages instead of cryptic timeouts?
• Circuit breakers & rate limiting: Do upstream controls activate correctly to prevent cascading failures?
• Automatic recovery: Can the system self-heal (e.g., pod restarts in Kubernetes) after load is reduced?
• Data integrity: Is there any corruption of the KV Cache or model state during overload?

This objective maps system performance to underlying hardware constraints. By correlating load with resource metrics, engineers can identify bottlenecks and right-size infrastructure.

• GPU Utilization: Does it plateau at 100%, indicating a compute-bound workload?
• Memory Pressure: Does GPU memory or system RAM exhaust before compute, indicating a memory-bound workload? This is critical for models using large KV Caches.
• I/O and Network: Do disk or network interfaces become saturated?
• CPU Contention: Is the CPU overwhelmed by pre/post-processing or tokenization, starving the GPU?

Under stress, average latency becomes less meaningful; the behavior of the slowest requests is paramount. This objective focuses on understanding and mitigating tail latency.

• Latency distribution: How do P90, P95, and P99 latencies scale as load increases? A sharp rise in P99 indicates queueing or resource contention issues.
• Determinants of tail latency: Is it caused by garbage collection pauses, cold starts for new model instances, or interference between concurrent requests?
• Impact of scheduling: How do techniques like continuous batching affect tail latency under overload?

Stress test data provides the empirical basis for setting realistic, data-driven operational limits. This translates technical limits into business policies.

• Maximum safe load: Defines the QPS threshold before SLOs are violated, informing auto-scaling rules.
• Alerting thresholds: Sets triggers for when latency (e.g., P95) or error rates approach dangerous levels.
• Capacity planning: Informs decisions on hardware procurement and cluster sizing based on projected growth.
• SLO Validation: Proves that Service Level Objectives (e.g., "P99 latency < 500ms") are achievable under defined load conditions.

Complex inference pipelines depend on multiple services (databases, feature stores, other models). Stress testing reveals fragility in these dependencies.

• Downstream service impact: Does overloading the model service cause failures in monitoring or logging systems?
• Cache effectiveness: How does performance degrade when KV Cache hit rates fall or when external caches (e.g., Redis) become saturated?
• Multi-tenant interference: In a shared cluster, can a stressed service degrade performance for other, unrelated models due to resource contention (noisy neighbor problem)?
• Configuration limits: Are there OS-level limits (e.g., file descriptors, network buffers) that become bottlenecks?

High-percentile latencies are the most critical stress test indicators, as they reveal the experience of the slowest requests when the system is saturated. Under stress, tail latency often increases exponentially rather than linearly, exposing bottlenecks like garbage collection pauses, memory swapping, or thread contention. Monitoring P99 latency is essential for understanding user-perceived degradation. A system may maintain acceptable median latency while its P99 spikes, indicating instability.

This metric identifies the absolute maximum queries per second (QPS) or tokens per second (TPS) the system can sustain before performance collapses. The goal is to plot a throughput-latency curve and locate the saturation point—the inflection where latency increases dramatically for minimal throughput gain. Stress tests measure the plateau of this curve, revealing the hard limit of the system's processing capacity, often constrained by GPU compute, memory bandwidth, or internal scheduling overhead.

As load exceeds design limits, systems fail in specific ways. Stress tests meticulously catalog:

• HTTP 5xx errors (e.g., 503 Service Unavailable) from overwhelmed servers.
• GPU out-of-memory (OOM) errors from exhausted VRAM.
• Request timeouts and dropped connections.
• Model-specific failures like generation of malformed outputs or hallucinations under load. Quantifying the error rate as a function of load is crucial for defining operational boundaries and implementing graceful degradation policies.

Stress tests determine if the system is compute-bound or memory-bound at its limits. Key utilization metrics include:

• GPU Utilization (%): Sustained near-100% utilization indicates compute saturation.
• GPU Memory Usage: Tracking consumption to the limit reveals memory-bound scenarios.
• CPU Utilization: High system CPU can indicate preprocessing, scheduling, or data movement overhead.
• Network/PCIe Bandwidth: Saturation can occur when shuffling large model weights or KV caches. Tools like NVIDIA Nsight Systems or nvtop are used for this profiling.

Under extreme load, requests compete for finite resources. Stress tests measure:

• Queue Length: The number of requests waiting for execution. A growing queue is a primary symptom of saturation.
• Scheduler Latency: The time a request spends waiting in a scheduler (e.g., for a continuous batching slot) before execution begins.
• Lock Contention: Increased time spent waiting for mutexes or other synchronization primitives. These metrics reveal inefficiencies in the serving architecture that are only visible under concurrent, heavy load.

A critical stress test phase is the ramp-down. Metrics monitored include:

• Latency Recovery Time: How long after peak load does latency return to baseline?
• Resource Leak Detection: Does memory or connection usage return to pre-test levels, or are there leaks?
• Service Restoration: Can the system automatically recover from transient failures (e.g., auto-scaling kicking in), or does it require manual intervention? This evaluates the resilience and operational robustness of the inference service beyond raw performance.

Load testing simulates expected or peak user traffic to measure performance under normal and high-load conditions. Unlike stress testing, which seeks the breaking point, load testing validates that the system meets Service Level Objectives (SLOs) under target load.

• Goal: Verify performance at specified concurrency levels.
• Method: Uses a synthetic workload or real-world workload replay.
• Output: Metrics like P99 latency and throughput at expected load.

These are the two primary metrics measured during any performance test.

• Latency: The time delay for a single request. Critical sub-metrics include:
  • Time to First Token (TTFT): Initial generation delay.
  • Time per Output Token (TPOT): Streaming speed.
  • Tail Latency (P99): Worst-case delays that impact user experience.
• Throughput: The system's processing rate, measured in Queries Per Second (QPS) or Tokens Per Second (TPS).

Stress testing maps how these metrics degrade as load exceeds capacity.

The systematic process of identifying the limiting resource in an inference pipeline during stress testing. A system is typically either:

• Compute-Bound: Limited by GPU/CPU FLOPs. Latency increases linearly with load until saturation.
• Memory-Bound: Limited by memory bandwidth (e.g., loading model weights). Performance plateaus early.

Tools like performance profilers and the Roofline Model help pinpoint bottlenecks in GPU kernels, memory copies, or I/O operations.

A Service Level Objective (SLO) is a target for system reliability or performance, such as "P99 latency < 200ms for 99.9% of requests." Stress testing directly informs SLO setting by revealing the failure boundary.

• Service Level Indicator (SLI): The measured metric (e.g., latency) used to evaluate an SLO.
• Use in Stress Testing: Determines the "safe" operating zone before the saturation point where SLOs are violated.

A performance baseline is a set of reference metrics (latency, throughput, resource use) established under controlled conditions. It is a prerequisite for meaningful stress testing.

• Establishing a Baseline: Done via initial load testing on known hardware/software.
• Performance Regression: An undesirable degradation from the baseline. Stress tests after system changes (model updates, new kernels) can reveal regressions in maximum capacity or efficiency.

The fundamental graph produced by incremental load and stress testing. It plots system throughput (QPS) on the X-axis against observed latency (e.g., P50, P99) on the Y-axis.

• Linear Region: Latency stable as throughput increases.
• Knee of the Curve: Latency begins to increase noticeably. This is often the target operating point.
• Saturation Point: Throughput plateaus at maximum capacity; latency increases sharply.
• Failure Point: System begins rejecting requests or errors spike.

This curve is the primary artifact for capacity planning.