Queries Per Second (QPS)

Queries Per Second (QPS) measures the number of complete inference requests a system can process in one second. It is a system-level throughput metric that aggregates the completion of all requests, regardless of their individual complexity or output length.

• Unit of Measurement: Requests/Second.
• Scope: Measures completed end-to-end requests, from receipt of input to delivery of final output.
• Contrast with Latency: While latency (e.g., P99) measures the time for a single request, QPS measures the volume the system can handle collectively. They exist in a trade-off relationship, visualized on a throughput-latency curve.

QPS does not exist in isolation; it must be interpreted alongside other key performance indicators (KPIs) to provide a complete picture.

• Tokens Per Second (TPS): For language models, TPS measures token generation rate. A system can have high QPS for short requests but lower TPS if requests are long. QPS * Avg Output Tokens ≈ TPS.
• Latency Percentiles (P50, P99): As QPS increases toward the system's saturation point, latency typically degrades, especially at the tail (P99). A target QPS must be evaluated against latency Service Level Objectives (SLOs).
• Hardware Utilization: High QPS with low GPU/CPU utilization indicates inefficiency, while QPS plateauing at 100% utilization identifies a compute-bound or memory-bound bottleneck.

A system's maximum achievable QPS is determined by a confluence of hardware, software, and workload factors.

• Model Architecture & Size: Larger models require more FLOPs per token, reducing potential QPS.
• Inference Optimization: Techniques like continuous batching, KV cache management, and model quantization directly increase QPS by improving hardware utilization and reducing compute per request.
• Hardware (GPU/CPU): Peak FLOPs and memory bandwidth define the theoretical QPS ceiling.
• Request Profile: Batch size, input/output sequence lengths, and the variance in these (load distribution) are primary determinants. Homogeneous, short requests yield higher QPS.
• System Overhead: Network I/O, pre/post-processing, and orchestration framework latency subtract from available compute time, capping effective QPS.

Measuring QPS accurately requires controlled, reproducible benchmarking that reflects production scenarios.

• Synthetic vs. Real-World Workloads: Benchmarks use synthetic workloads for consistency, but final validation requires real-world workload patterns.
• Establishing a Baseline: Measure QPS under a target latency SLO (e.g., QPS@P99<200ms) to create a performance baseline.
• Load Testing: Gradually increase request rate to plot the throughput-latency curve and identify the saturation point where latency spikes.
• Industry Standards: Benchmarks like MLPerf Inference provide standardized models and datasets for fair QPS comparison across systems.
• Concurrent Requests: QPS is achieved by processing concurrent requests. Testing must simulate appropriate concurrency levels to expose resource contention.

For CTOs and engineering managers, QPS is a direct input into infrastructure cost models and capacity plans.

• Cost per Query: Total inference cost divided by total queries over a period. Optimizing to achieve higher QPS on the same hardware reduces cost per query.
• Capacity Forecasting: Required QPS targets (from product forecasts) dictate the number of instances or GPUs needed. Planning must account for peak load, not just average.
• Scalability Analysis: QPS should scale near-linearly with added replicas (horizontal scaling) until bottlenecked by shared resources like load balancers or data planes.
• Budgeting: QPS projections translate directly to cloud compute or hardware procurement budgets under inference cost optimization initiatives.

Misinterpreting QPS can lead to poor architectural decisions and unmet performance expectations.

• Ignoring Tail Latency: Maximizing QPS without regard for P99 latency leads to poor user experience during traffic spikes.
• Overlooking Warm-Up: Reporting QPS during steady-state performance without accounting for cold start latency gives an incomplete view of production readiness.
• Homogeneous Assumption: A QPS number is only valid for a specific request profile. Changing average input length by 20% can change QPS by 40%.
• Not a Quality Metric: High QPS does not indicate accurate or correct model outputs. It is purely a measure of system throughput.
• The "Theoretical Max" Fallacy: Peak hardware FLOPs rarely translate directly to achievable QPS due to software overheads, memory bottlenecks, and operator inefficiency.

Latency is the time delay between submitting an inference request and receiving the complete response, measured in milliseconds. It is the primary metric for user-perceived responsiveness.

• Direct Trade-off with QPS: As QPS increases, latency typically rises due to resource contention and queueing.
• Types: Includes Time to First Token (TTFT) for streaming responses and end-to-end latency for complete requests.
• Critical for UX: High latency directly impacts application usability, even if QPS is high.

Time to First Token measures the latency from request submission until the first output token is generated by an autoregressive model. It represents the initial computational overhead.

• Components: Includes prompt encoding, model computation through all layers for the first position, and network transmission.
• User Perception: Crucial for streaming interfaces, as it determines the perceived 'startup time' of a response.
• Independent of Output Length: Unlike total latency, TTFT is not directly affected by the number of tokens generated.

Tokens Per Second is a throughput metric specific to language models that counts the total number of output tokens generated across all requests in one second.

• Relationship to QPS: QPS * (Average Output Tokens per Request) = TPS. A system can have high QPS with low TPS if requests are short.
• Compute Cost Proxy: More accurately reflects the computational load of text generation than QPS alone, as generating 100 tokens costs far more than generating 10.
• Key for Costing: Essential for estimating inference cost, which is often token-based.

Tail Latency refers to the high-percentile latencies (e.g., P95, P99) in a request distribution, representing the slowest requests.

• System Health Indicator: While average latency might be good, high P99 latency indicates instability, resource contention, or garbage collection pauses affecting some users.
• Determines SLOs: Service Level Objectives are often defined around P99 latency, not averages.
• Impact on QPS: A system with poor tail latency may need to operate at a lower QPS to meet its SLOs, as queueing exacerbates tail effects.

Concurrent Requests are multiple inference queries being processed simultaneously by a system. Managing concurrency is key to achieving high QPS.

• Parallelism vs. Queueing: Systems use techniques like continuous batching to process requests in parallel on hardware, maximizing QPS. Excess concurrency leads to queueing and increased latency.
• Resource Management: Each concurrent request consumes GPU/CPU memory and compute slices. The maximum supported concurrency is a fundamental system limit.
• Measurement: The number of requests 'in-flight' at any given moment, which directly influences achieved QPS and latency.

A Throughput-Latency Curve is a graph plotting system throughput (QPS) against request latency as load increases. It is the essential model for understanding performance limits.

• Three Key Regions:
  • Latency-Stable: Low load, latency is constant as QPS increases.
  • Knee of the Curve: Latency begins to increase noticeably with added QPS.
  • Saturation: System reaches maximum QPS; latency increases dramatically with any additional load.
• Operational Point: Defines the target QPS for a given latency SLO. Operating near the 'knee' maximizes utilization without violating SLOs.
• Benchmarking Tool: Generated via load testing to determine system capacity and optimal configuration.