Saturation Point

The saturation point marks the absolute maximum Queries Per Second (QPS) or Tokens Per Second (TPS) a system can sustain. Beyond this point, the system cannot process requests any faster, regardless of increased incoming load.

• This is the peak of the throughput-latency curve.
• It is a hard limit determined by the slowest component in the inference pipeline (e.g., GPU compute, memory bandwidth, CPU pre/post-processing).
• Example: A system may saturate at 1000 QPS; sending 1100 QPS will not result in 1100 QPS of output.

As load approaches the saturation point, latency begins to increase super-linearly. This is the primary operational signal that a system is nearing its limit.

• At low load, latency is stable and determined by single-request processing time.
• Near saturation, queueing delays dominate as requests wait for resources.
• Tail latency (P95, P99) spikes dramatically, often violating Service Level Objectives (SLOs) long before average throughput drops.
• This characteristic makes it dangerous to operate a system at precisely its measured saturation point.

Saturation is caused by the exhaustion of a critical, shared resource, leading to resource contention. The bottleneck resource defines the system's capacity.

• Common Bottlenecks:
  • GPU Compute: A compute-bound workload saturates the processor's FLOPs.
  • GPU Memory Bandwidth: A memory-bound workload saturates data transfer to/from VRAM.
  • CPU: Saturation in pre-processing (tokenization) or post-processing (detokenization).
  • KV Cache Memory: Running out of memory for attention keys and values in long-context scenarios.
• Identifying the specific bottleneck is the goal of bottleneck analysis.

Once the arrival rate of requests exceeds the system's maximum service rate at the saturation point, a request queue forms and grows indefinitely. This is a defining characteristic of a saturated system.

• Queue length is a direct function of the difference between arrival rate and service rate.
• Queues can form in software (e.g., in the model server) or in hardware (e.g., GPU command queues).
• Unbounded queue growth leads to unbounded latency and eventual system failure (e.g., out-of-memory errors).
• Techniques like load shedding or autoscaling are required to prevent this state.

On a throughput-latency curve, the throughput axis forms a distinct plateau at the saturation point. Increasing the number of concurrent requests does not increase measured throughput.

• This plateau represents the system's maximum capacity.
• The value of this plateau is the key metric for capacity planning and cost-per-inference calculations.
• The plateau's shape can be sharp (for rigid systems) or gradual (for systems with dynamic scheduling like continuous batching).
• Operating on the plateau is inefficient, as latency is high and user experience is poor, despite maximal resource usage.

As a system operates at or beyond its saturation point, the probability of request failures increases significantly due to timeouts and resource exhaustion.

• Client-side timeouts occur when latency exceeds a client's wait threshold.
• Server-side errors (e.g., 503 Service Unavailable, out-of-memory) increase as resources are overwhelmed.
• This directly impacts the Service Level Indicator (SLI) for error rate, breaking Service Level Objectives (SLOs).
• In stress testing, observing the rise in error rates is as important as measuring the latency blow-up.

Throughput is the rate at which an inference system processes requests, measured in queries per second (QPS) or tokens per second (TPS). It represents the system's maximum processing capacity before hitting the saturation point. Key distinctions:

• QPS: Measures complete request processing.
• TPS: Measures raw token generation speed for language models. Throughput is a primary indicator of a system's scale and economic efficiency.

Latency is the time delay between submitting an inference request and receiving the response. It is the user-facing metric most directly impacted as load approaches the saturation point. Critical sub-metrics include:

• Time to First Token (TTFT): Initial processing delay.
• Time per Output Token (TPOT): Incremental generation time.
• Tail Latency (P95/P99): The slowest requests, which degrade most severely at saturation.

Tail latency refers to the high-percentile latencies (e.g., P95, P99) in a request distribution. These represent the slowest requests a user might experience. As a system approaches its saturation point, tail latency increases dramatically due to resource contention and queueing effects. Managing tail latency is crucial for consistent user experience and meeting Service Level Objectives (SLOs).

A throughput-latency curve is the fundamental graphical model for identifying the saturation point. It plots system throughput against average or tail latency.

• Linear Region: Latency is stable as throughput increases.
• Knee of the Curve: Latency begins to increase noticeably; the system is nearing saturation.
• Saturation Region: Throughput plateaus at its maximum while latency increases sharply. This curve is essential for capacity planning and setting operational targets.

A Service Level Objective (SLO) is a target for system reliability or performance, such as "P99 latency < 200ms." The saturation point defines the maximum load a system can handle while still meeting its SLOs. Exceeding the saturation point causes SLO violations. SLOs are measured using Service Level Indicators (SLIs) like latency and error rate.

Resource contention occurs when multiple concurrent inference requests compete for finite system resources like GPU memory, compute cores, or memory bandwidth. It is the primary physical mechanism causing performance degradation at the saturation point. Symptoms include:

• Increased queueing delays.
• Cache thrashing.
• Inefficient hardware utilization. Mitigation involves optimization techniques like continuous batching and KV cache management.