Throughput-Latency Curve

The curve visualizes the inverse relationship between throughput and latency. As offered load increases, throughput rises linearly in the under-loaded region until it plateaus at the system's maximum capacity. Concurrently, latency remains low and stable until throughput approaches this capacity, after which it increases non-linearly due to queueing effects. This defines the primary trade-off: maximizing throughput forces acceptance of higher latency, and minimizing latency requires operating below peak capacity.

The knee point is a critical region where latency begins to increase noticeably while throughput gains diminish. It represents the optimal operating point for many production systems, balancing resource utilization with acceptable response times.

• Identifying the Knee: It is often found where the latency curve's slope increases sharply, or where throughput growth slows.
• Operational Significance: Operating just before the knee provides headroom for traffic spikes without severe latency degradation. Exceeding it risks entering the saturation region where small load increases cause large latency jumps.

Beyond the knee, the system enters saturation. Throughput reaches a hard ceiling—the saturation point—determined by the system's bottleneck resource (e.g., GPU FLOPs, memory bandwidth).

• Latency Explosion: In saturation, incoming requests queue, causing latency to rise asymptotically. The curve becomes nearly vertical.
• System Collapse: Under extreme overload, latency can become unbounded, and effective throughput may drop due to thrashing or timeouts, leading to service failure. This highlights the danger of operating in the deep saturation region.

Batching and concurrency management directly reshape the curve. Dynamic batching groups requests to improve GPU utilization, shifting the throughput ceiling higher but often increasing tail latency due to batch formation delays.

• Larger Batches: Increase maximum throughput but shift the latency curve upward, trading per-request speed for aggregate efficiency.
• Concurrent Queries: Managing many simultaneous requests can reveal resource contention (e.g., memory bandwidth, cache thrashing) that lowers the saturation point and worsens latency, creating a less favorable curve.

For LLM and AI inference, the curve is uniquely influenced by several factors:

• Prompt vs. Generation: Long prompts increase Time to First Token (TTFT), raising the baseline latency. Long generations are constrained by Time per Output Token (TPOT).
• KV Cache Management: Efficient cache usage increases throughput capacity. Poor management leads to memory bottlenecks, lowering the saturation point.
• Model Architecture: Mixture of Experts (MoE) models have flatter latency curves under load due to sparse activation, while dense models show a sharper knee.
• Hardware Utilization: The curve flattens when the system becomes compute-bound versus steepening when memory-bound.

The curve is a practical tool for setting Service Level Objectives (SLOs) and planning scale. By plotting latency percentiles (P50, P99) against throughput, engineers can define the maximum load that keeps tail latency within SLO bounds.

• Horizontal Scaling: Adding replicas shifts the entire throughput axis, allowing more total load at the same latency.
• Autoscaling Triggers: Latency metrics derived from the curve (e.g., "scale when P95 latency > X ms") are more effective than raw utilization metrics.
• Capacity Planning: The saturation point provides a clear target for maximum load per instance, informing cluster sizing and cost forecasting.

The curve is used to determine the operational sweet spot for a deployed model. Engineers analyze the curve to set target load levels that maintain acceptable latency (e.g., P99 < 200ms) while maximizing throughput. This directly informs decisions on autoscaling policies, instance count, and hardware provisioning to meet Service Level Objectives (SLOs) without over-provisioning resources.

• Key Action: Identify the knee of the curve, where latency begins to increase non-linearly, as the maximum sustainable load.
• Example: A system may handle 1000 QPS with 50ms latency, but at 1200 QPS latency jumps to 500ms, indicating the saturation point.

The shape of the curve reveals the nature of system bottlenecks. A sharp latency increase at low throughput suggests a memory-bound or I/O-bound system (e.g., constrained by memory bandwidth or model loading). A more gradual increase points to a compute-bound system (limited by FLOPs).

• Memory-Bound Signal: Throughput plateaus quickly while latency rises steeply.
• Compute-Bound Signal: Throughput scales more linearly with load before saturating.
• Optimization Path: This diagnosis guides engineers toward techniques like operator fusion, KV cache optimization, or moving to higher-bandwidth memory to reshape the curve.

Curves provide an apples-to-apples comparison for different optimization techniques, hardware, or model versions. By plotting curves for a baseline and an optimized system (e.g., with quantization or continuous batching), engineers can quantify the improvement in maximum throughput at a target latency, or the latency reduction at a target throughput.

• A/B Testing: Essential for evaluating the impact of new inference engines, compiler flags, or GPU memory optimization strategies.
• Vendor Selection: Used to compare cloud instance types or accelerator cards based on their performance under load, not just peak theoretical specs.

Generating the full curve is the core of load testing. Engineers use tools to incrementally increase request rates (Queries Per Second) while measuring latency percentiles (P50, P90, P99). This process identifies the saturation point and system behavior under overload.

• Stress Testing: Pushing beyond saturation reveals failure modes and tail latency behavior.
• Real-World Simulation: Using synthetic workloads that mimic production traffic patterns (e.g., varying input lengths) creates a realistic curve for capacity planning.
• Objective: To ensure the system can handle peak traffic while staying within SLOs.

For CTOs and engineering managers, the curve translates technical performance into business metrics. By overlaying infrastructure cost data, they can calculate the inference cost per request at different points on the curve.

• Optimization Goal: Find the load point that minimizes total cost while meeting latency SLOs. Running a system too far below saturation wastes resources; running too close risks SLO violations.
• Decision Framework: Answers questions like: "Is it cheaper to use a larger, more efficient instance at higher utilization, or more smaller instances?" The curve provides the performance data needed for this analysis.

In production, systems use the curve's principles to implement intelligent request management. If latency metrics indicate the system is approaching the knee of the curve, an admission controller can begin queueing or shedding low-priority requests to protect latency for high-priority ones.

• Application: Concurrent request scheduling in systems using continuous batching is tuned based on the expected latency impact of increasing batch size (which increases throughput but also latency).
• Goal: To operate stably near the optimal point on the curve, preventing a cascade of delays caused by resource contention and queueing.

Latency is the time delay between a request being submitted to an inference system and the corresponding response being received, typically measured in milliseconds. It is the primary user-facing performance metric. For autoregressive models like LLMs, latency is often decomposed into Time to First Token (TTFT) and Time per Output Token (TPOT). High latency degrades user experience, while low latency is critical for interactive applications.

Throughput is the rate at which an inference system processes requests, commonly measured in queries per second (QPS) or tokens per second (TPS). It represents the system's capacity or batch processing efficiency. Throughput and latency exist in a fundamental trade-off: increasing throughput (by batching more requests) typically increases average latency. The goal of optimization is to maximize throughput while keeping latency within acceptable Service Level Objectives (SLOs).

Tail latency refers to the high-percentile latencies (typically P95, P99) in a request distribution, representing the slowest requests. While average latency might be low, tail latency determines user-perceived performance for the worst-case scenarios. It is critical for Service Level Agreements (SLAs). Causes include:

• Resource contention (e.g., memory bandwidth saturation)
• Garbage collection pauses
• Noisy neighbors in multi-tenant systems
• Variability in request complexity Mitigating tail latency is a key focus of production inference engineering.

The saturation point is the load level at which an inference system reaches its maximum throughput capacity. On a throughput-latency curve, this is the 'knee' where latency begins to increase exponentially. Beyond this point:

• Additional requests queue
• Latency rises sharply
• System may become unstable Identifying the saturation point through load testing and stress testing is essential for defining safe operating limits and scaling triggers.

A Service Level Objective is a specific, measurable target for system performance that forms the basis of an operational agreement. For inference systems, SLOs are often defined using latency and throughput metrics, such as "P99 latency < 200ms" or "throughput > 100 QPS." The throughput-latency curve is used to validate that the system can meet its SLOs under expected load. SLOs are monitored via Service Level Indicators (SLIs) like request latency and error rate.

Bottleneck analysis is the systematic investigation to identify the component limiting overall inference performance. The shape of the throughput-latency curve often reveals the bottleneck type:

• Compute-bound: Latency rises slowly with load until a hard compute limit.
• Memory-bound: Performance degrades due to memory bandwidth limits, often seen with large models.
• I/O-bound: Limited by data loading (e.g., disk, network). Tools like performance profilers and models like the Roofline Model are used to diagnose bottlenecks and guide optimization efforts in GPU memory, kernel execution, or scheduling.