Queries Per Second (QPS)

QPS and latency are intrinsically linked. A system's throughput-latency curve shows that as QPS increases, average and tail latency (P99/P95) typically rise due to resource contention and request queuing delay. The optimal operating point is the highest sustainable QPS before latency exceeds the Service Level Objective (SLO). Increasing QPS beyond this point causes latency to spike exponentially.

A QPS value is meaningless without a corresponding latency target. A valid specification is: '500 QPS at P99 latency < 200ms'. This means the system can handle 500 requests per second while ensuring 99% of requests complete within 200ms. The SLO for latency defines the quality of service. Measuring QPS without monitoring latency leads to a degraded user experience under load.

QPS is a measure of load. It is directly influenced by the number of concurrent requests the system is processing. Key factors determining achievable QPS include:

• Hardware Capacity: GPU memory bandwidth, CPU cores, and network I/O.
• Model Characteristics: Size, architecture, and computational graph.
• Optimization Techniques: Use of continuous batching, model quantization (INT8/FP16), and efficient kernels via TensorRT.
• Request Profile: Payload size, input/output token length, and complexity.

QPS is the aggregate result of multiple underlying performance factors. It cannot be optimized in isolation. Improving QPS requires addressing specific bottlenecks:

• Inference Latency: Reducing prefilling and decoding latency.
• Memory Efficiency: Using PagedAttention (as in vLLM) to reduce KV cache waste.
• Overhead Reduction: Minimizing GPU kernel launch overhead through operator fusion.
• System Lag: Accounting for autoscaling lag and cold start latency during traffic fluctuations.

QPS is the fundamental metric for infrastructure sizing and cost forecasting. Engineers use it to:

• Determine the number of GPU instances required to meet expected traffic.
• Set scaling rules for cloud deployments.
• Calculate the cost per query for a deployed model.
• Establish performance baselines and detect regressions via canary analysis. It translates business demand (user requests) into technical resource requirements.

Accurate QPS measurement requires load testing that mimics production. This includes:

• Variable request patterns (bursts, sustained load).
• Realistic input lengths and payloads.
• A mix of synchronous and asynchronous inference patterns.
• Monitoring of both end-to-end latency (user perspective) and time to first token (TTFT) for streaming.
• Tools like profiling (CPU/GPU) and distributed tracing are essential to move from measuring QPS to understanding the bottleneck identification that limits it.

The raw computational capacity of the underlying hardware is the fundamental ceiling for QPS. Key factors include:

• GPU/Accelerator Throughput: The FLOPs (Floating-Point Operations per Second) and memory bandwidth of the inference accelerator (e.g., NVIDIA H100, A100) directly limit how many model forward passes can be executed per second.
• CPU & Memory: The host CPU speed and system RAM bandwidth affect pre/post-processing, tokenization, and orchestration overhead, which can bottleneck the accelerator.
• Network Interface: For distributed systems, the bandwidth and latency of the network card (e.g., NVLink, InfiniBand) govern how quickly data can be sharded or aggregated across nodes.

The intrinsic complexity of the model being served is a primary determinant of per-request compute cost.

• Parameter Count & Layers: Larger models (e.g., 70B+ parameters) require more computations and memory transfers per token, reducing potential QPS compared to smaller models (e.g., 7B parameters) on the same hardware.
• Attention Mechanism: The quadratic complexity of standard attention with sequence length is a major bottleneck. Architectures with linear or sparse attention (e.g., MQA, GQA) can significantly improve QPS for long contexts.
• Activation Memory: The size of intermediate activations during inference impacts memory bandwidth pressure and cache efficiency.

Software-level optimizations dramatically increase QPS by improving hardware utilization.

• Continuous Batching: Dynamically batches incoming requests of varying lengths, keeping the GPU saturated even as individual requests finish, often increasing throughput 5-10x over static batching.
• Model Quantization: Reducing weight and activation precision from FP32 to FP16, INT8, or INT4 (e.g., via GPTQ, AWQ) cuts memory footprint and increases compute speed on supported hardware, directly boosting QPS.
• Kernel Fusion & Graph Optimization: Compilers like TensorRT or ONNX Runtime fuse sequential operations into single, optimized GPU kernels, reducing launch overhead and memory I/O.
• Speculative Decoding: Uses a small draft model to propose token sequences verified in parallel by the main model, reducing the number of slow autoregressive steps and improving QPS for longer outputs.

Efficient memory usage determines how many concurrent requests can be handled.

• KV Cache Efficiency: The Key-Value cache for autoregressive models consumes vast memory. PagedAttention (as used in vLLM) eliminates fragmentation by managing the KV cache in non-contiguous blocks, allowing much higher concurrency and QPS.
• Static vs. Dynamic Shapes: Systems that can handle variable sequence lengths without recompilation (dynamic shapes) are more flexible but may sacrifice some peak QPS compared to pre-compiled static shapes.
• CPU-GPU Data Transfer: Minimizing the movement of data across the PCIe bus (e.g., by keeping tokenizers on GPU) reduces per-request overhead.

The design of the inference server and its scheduler dictates how work is parallelized.

• Request Scheduling Policy: Policies like First-In-First-Out (FIFO) or shortest-job-first affect queuing delays and fairness, influencing the throughput-latency trade-off.
• Concurrent Request Limit: The maximum number of requests processed simultaneously is tuned to maximize GPU utilization without causing excessive request queuing delay or memory overflow.
• Autoscaling & Load Balancing: The speed and efficiency with which a cluster can scale replicas in response to load (autoscaling lag) determines the system's ability to maintain QPS during traffic spikes.

QPS cannot be evaluated in isolation; it exists on a throughput-latency curve. Pushing for maximum QPS by overloading the system with concurrent requests will inevitably increase average and tail latency (P99/P95).

• Operating Point Selection: The target QPS is chosen based on a Service Level Objective (SLO) for Latency (e.g., P99 < 1s). The system is provisioned and tuned to operate at the QPS that meets this SLO.
• Performance Baseline: Establishing a baseline under a target load is essential for detecting regressions. Tools like profiling (CPU/GPU) and distributed tracing are used for bottleneck identification to optimize this trade-off.
• Canary Analysis: New model versions or configurations are tested against the baseline QPS/latency on a subset of traffic before full deployment.

Inference latency is the total time delay between submitting an input to a machine learning model and receiving its output. It is the foundational constraint against which QPS is measured. A system's maximum sustainable QPS is the point where adding more queries causes latency to exceed a defined Service Level Objective (SLO).

• Components: Includes pre-processing, model forward pass, post-processing, and any network transfer.
• Primary Trade-off: Systems are typically tuned to maximize QPS while keeping latency (especially tail latency) below an acceptable threshold.

A throughput-latency curve is a graph that plots the relationship between a system's request throughput (QPS) and its corresponding average or percentile latency. It is the essential tool for capacity planning and performance optimization.

• Shape: Latency remains low and stable until a saturation point, after which it increases exponentially as queues form.
• Use Case: Engineers use this curve to identify the optimal operating point—the highest QPS achievable before latency degrades unacceptably. This defines the practical QPS limit.

Concurrent requests are the number of client inference queries being processed simultaneously by a serving system. It is a direct driver of resource utilization and a key factor linking QPS and latency.

• Relationship to QPS: QPS = Concurrent Requests / Average Latency (Little's Law).
• Impact: High concurrency increases GPU utilization but can lead to request queuing delay if it exceeds the system's parallel processing capacity, directly increasing latency.

A Service Level Objective (SLO) for latency is a target reliability goal defined for a specific latency percentile (e.g., P99 < 200ms). It is the contractual performance benchmark that a system's QPS must satisfy.

• Defines QPS Validity: A reported QPS figure is meaningless without an associated latency SLO. A system might handle 1000 QPS at P50 latency but only 200 QPS at a strict P99 latency SLO.
• Error Budgets: SLOs create error budgets for performance, guiding decisions on autoscaling, deployment strategies like canary analysis, and infrastructure investment.

Continuous batching (or dynamic batching) is a critical inference optimization technique that directly maximizes QPS for a given latency target. It dynamically adds new requests to a running batch on the GPU as previous requests finish generation.

• Efficiency vs. Static Batching: Eliminates the need to wait for a fixed batch to complete, dramatically improving GPU utilization and throughput.
• Latency Benefit: Reduces average latency for individual requests compared to waiting for a full static batch, allowing higher QPS within latency SLOs. Engines like vLLM implement this.

Tail latency refers to the high-percentile response times (e.g., P95, P99) that represent the slowest requests in a distribution. Managing tail latency is paramount for defining a realistic, user-experience-focused QPS.

• Importance: While average latency might be low, a high P99 latency means 1% of users have a poor experience. QPS must be capped to keep tail latency in check.
• Causes: Often caused by variance in request queuing delay, cold start latency, garbage collection, or network interference. Optimization focuses on reducing variance, not just average speed.