Throughput

Queries Per Second (QPS) measures the number of complete inference requests a system can process in one second. It is a holistic, end-to-end metric for request-level throughput.

• Use Case: Best for measuring overall system capacity for request/response APIs, where each query (e.g., a classification task, a short text completion) is a discrete unit of work.
• Limitation: QPS does not account for variable request complexity or output size. A system with high QPS for simple requests may struggle with longer, more complex prompts.

Tokens Per Second (TPS) measures the total number of output tokens generated across all active requests in one second. It is the fundamental throughput metric for autoregressive text generation models.

• Calculation: TPS = (Total Output Tokens Generated) / (Measurement Window).
• Key Insight: TPS directly correlates with GPU compute utilization during the generation phase. Optimizations like continuous batching aim to maximize this metric.
• Context: For a fixed model, TPS is influenced by batch size, sequence length, and the efficiency of the KV cache.

Inferences Per Second (IPS) is a general throughput metric representing the number of model forward passes completed per second. It applies broadly across model types, not just language models.

• Use Case: Common for non-autoregressive tasks like computer vision (image classification, object detection) or embedding generation, where each input yields one output.
• Relationship to QPS: For single-pass models, IPS and QPS are often equivalent. For generative models, one "query" may require many sequential "inferences" (steps) to produce the full output.

Throughput and latency have a non-linear relationship defined by the throughput-latency curve. Increasing load (to raise throughput) eventually causes queueing delays, increasing latency.

• Underloaded System: Latency is stable and low; throughput increases linearly with concurrent requests.
• At the Knee: The system approaches its saturation point. Latency begins to increase noticeably.
• Overloaded System: Throughput plateaus at its maximum while latency grows exponentially due to queueing.
• Engineering Goal: Operate just before the knee of the curve to maximize throughput while adhering to latency SLOs.

Hardware Utilization measures the percentage of available computational resources actively used during inference. High throughput requires high utilization, but the relationship is not always linear.

• GPU Utilization: Percentage of streaming multiprocessors (SMs) actively executing kernels. Aim for >90% during steady-state inference.
• Memory Bandwidth Utilization: Critical for memory-bound workloads. Low utilization here can indicate inefficient memory access patterns.
• The Roofline Model: This analytical tool helps determine if a workload/kernel is compute-bound (limited by FLOPs) or memory-bound (limited by data movement). True maximum throughput is achieved when the system operates at the "roofline."

The Saturation Point is the maximum sustainable throughput a single instance can deliver. Beyond this point, tail latency spikes and request failures may occur.

• Identification: Found via load testing and stress testing, observing where the throughput-latency curve flattens.
• Horizontal Scaling: To increase total system throughput beyond a single node's saturation, instances are added behind a load balancer. Total throughput scales near-linearly if the workload is stateless and there is no resource contention for shared services.
• Vertical Scaling: Increasing the resources (e.g., a larger GPU) of a single instance can raise its individual saturation point, often a simpler initial optimization.

A dynamic scheduling technique that groups multiple inference requests (batches) together for parallel execution on the GPU. Unlike static batching, it continuously adds new requests to the batch as others finish, maximizing GPU utilization and overall system throughput. This is critical for handling variable-length requests, like chat completions, where users don't all respond at the same time.

• Key Benefit: Eliminates idle GPU time by keeping computational units saturated.
• Implementation: Managed by advanced inference servers like vLLM or NVIDIA Triton.

Transformer-based models (LLMs) store computed Key-Value (KV) pairs during the generation of previous tokens to avoid recomputation. Efficient management of this KV Cache is paramount for throughput.

• Memory Bottleneck: The cache grows linearly with batch size and sequence length, becoming the primary memory constraint.
• Techniques: PagedAttention (used in vLLM) treats the cache like virtual memory, allowing non-contiguous storage and drastically reducing memory fragmentation.
• Impact: Enables much larger effective batch sizes, directly increasing tokens/second throughput.

The process of reducing the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point FP32 to 8-bit integers INT8). This directly boosts throughput by:

• Reducing Memory Bandwidth Pressure: Lower precision weights require less data movement, alleviating a common memory-bound bottleneck.
• Increasing Compute Speed: Many hardware accelerators (GPUs, NPUs) have specialized units for low-precision arithmetic that perform more operations per second.
• Trade-off: Requires careful calibration to minimize accuracy loss, using techniques like GPTQ or AWQ for post-training quantization.

A technique that uses a small, fast draft model to predict a sequence of several future tokens. These speculative tokens are then verified in parallel by the larger, accurate target model in a single forward pass.

• Throughput Gain: Accepts multiple tokens per expensive target model run, increasing overall tokens per second (TPS).
• Condition: Effective when the draft model has high prediction accuracy (acceptance rate).
• Use Case: Ideal for increasing throughput in latency-sensitive, auto-regressive decoding scenarios.

A low-level optimization that combines multiple sequential computational operations (e.g., matrix multiply followed by an activation function like GeLU) into a single, custom fused kernel.

• Reduces Overhead: Eliminates intermediate memory writes and reads between operations, which are costly.
• Improves Hardware Utilization: Creates larger, more efficient workloads for GPU streaming multiprocessors.
• Implementation: Done via deep learning compilers like Apache TVM or MLIR, and is a core optimization in frameworks like PyTorch's torch.compile.

For sparse Mixture of Experts models, throughput is optimized by efficient conditional computation. Only a subset of the model's total parameters (the 'experts') are activated per token.

• Throughput Advantage: Enables extremely large model capacities (e.g., 1T+ parameters) without a proportional increase in compute cost per token.
• Challenge: Requires sophisticated routing logic and communication to manage which experts are loaded into memory, making batch sizing and load balancing critical.
• Systems Work: High throughput hinges on minimizing the overhead of the gating network and expert swapping.

Latency is the time delay between a request being submitted to an inference system and the corresponding response being received, typically measured in milliseconds. It represents the user-perceived delay, whereas throughput measures system capacity.

• Direct Trade-off: Systems often exhibit a throughput-latency curve, where pushing for higher throughput increases average latency due to queuing and resource contention.
• Key Types: Includes Time to First Token (TTFT) for initial response and Time per Output Token (TPOT) for generation speed.
• Measurement: Critical percentiles like P99 latency (tail latency) determine service reliability.

Queries Per Second is a fundamental throughput metric measuring the number of complete inference requests a system can process in one second. It is agnostic to the complexity or output length of individual requests.

• System-Level Metric: QPS measures overall system capacity for request processing.
• Contrast with TPS: Unlike Tokens Per Second (TPS), QPS counts entire requests, making it suitable for classification or embedding models where output size is fixed.
• Load Testing: Used to define a system's saturation point and maximum operational capacity.

Tokens Per Second is a throughput metric specific to autoregressive language models, measuring the total number of output tokens generated across all requests in one second. It directly correlates with the computational cost of text generation.

• Generation Speed: Reflects the efficiency of the model's forward passes and KV cache management.
• Influenced by Batching: Techniques like continuous batching maximize TPS by keeping GPU compute units saturated.
• User Experience: High TPS reduces wait time for long completions, complementing low TTFT and TPOT.

Hardware Utilization measures the percentage of available computational resources—such as GPU cores, tensor cores, or memory bandwidth—that are actively engaged during inference. High utilization is essential for achieving maximum throughput.

• Efficiency Indicator: Low GPU utilization often signifies a memory-bound workload or inefficient scheduling.
• Roofline Model: This model analyzes if performance is limited by peak compute (compute-bound) or by memory bandwidth.
• Optimization Target: Techniques like operator fusion and mixed precision inference aim to push utilization closer to 100% to maximize throughput.

Concurrent Requests are multiple inference queries being processed simultaneously by a system. The ability to handle high concurrency is key to achieving high throughput in multi-user serving environments.

• Parallel Execution: Managed by inference servers (e.g., TensorFlow Serving, vLLM, Triton) to batch requests dynamically.
• Resource Contention: High concurrency can lead to contention for GPU memory, CPU, or I/O, increasing tail latency if not managed properly.
• Scheduling: Advanced schedulers use continuous batching to group requests of varying lengths efficiently, maximizing throughput without starving individual queries.

A Service Level Objective is a specific, measurable target for system performance or reliability that forms the basis of an operational agreement. For throughput, SLOs often define minimum acceptable rates under specific conditions.

• Performance Guarantee: Example: "The service shall sustain 1000 QPS with P99 latency < 500ms."
• Based on SLIs: Defined using Service Level Indicators like throughput (QPS/TPS) and latency percentiles.
• Engineering Driver: SLOs dictate infrastructure scaling, model optimization (e.g., quantization), and bottleneck analysis to ensure targets are met cost-effectively.