Tokens Per Second (TPS)

Tokens Per Second (TPS) is calculated as the total number of output tokens generated divided by the total wall-clock time measured in seconds. The formula is:

TPS = (Total Output Tokens) / (Measurement Window in Seconds)

• Total Output Tokens: Sum of all tokens generated for all completed requests during the measurement period.
• Measurement Window: A sustained period of steady-state inference, excluding cold-start phases.
• This differs from Time per Output Token (TPOT), which is a latency metric (seconds/token). High TPS requires optimizing both low TPOT and efficient concurrent processing.

TPS is intrinsically linked to latency metrics and the level of request concurrency. Under ideal conditions, the relationship can be approximated by Little's Law: TPS ≈ Concurrency / TPOT.

• High Concurrency, Fixed TPOT: Increasing the number of concurrent requests typically increases TPS until a system bottleneck (e.g., memory bandwidth) is reached.
• Latency-Tradeoff: As concurrency increases to boost TPS, Tail Latency (P99) often degrades due to resource contention and queueing delays.
• Saturation Point: The maximum achievable TPS occurs at the system's saturation point, beyond which latency increases exponentially without TPS gains. Performance profiling involves tracing the Throughput-Latency Curve to identify this optimal operating point.

Achieving high TPS requires eliminating sequential bottlenecks in the inference pipeline. The major constraints are:

• Memory Bandwidth (Memory-Bound): The dominant bottleneck for autoregressive decoding. Each generated token requires loading the model's Key-Value (KV) Cache and weights from GPU memory. Techniques like KV Cache Management and quantization aim to reduce this pressure.
• Compute Capacity (Compute-Bound): For very large models or complex operations, the limit becomes the processor's FLOPs. This is analyzed using models like the Roofline Model.
• Scheduling Overhead: Inefficient Continuous Batching can lead to GPU idle time, directly capping TPS. Optimal batching dynamically groups requests to maximize GPU Utilization.
• I/O and Pre/Post-Processing: Tokenization, detokenization, and network serialization can become bottlenecks at very high TPS, shifting the limit from the GPU to the CPU or network.

Maximizing TPS is the goal of Inference Optimization. Key techniques include:

• Continuous Batching: Dynamically groups requests of varying lengths to minimize padding and keep GPU compute units saturated, dramatically improving TPS over static batching.
• KV Cache Optimization: Methods like paged attention manage the cache efficiently to support more concurrent sequences, directly increasing TPS.
• Model Quantization: Reducing weight precision (e.g., to FP16, INT8) via Post-Training Quantization decreases memory bandwidth consumption and increases compute efficiency, boosting TPS.
• Speculative Decoding: Uses a small draft model to propose multiple tokens which are then verified in parallel by the large target model, effectively reducing the average TPOT and increasing TPS.
• Operator/Kernel Fusion: Combines multiple low-level operations into a single optimized GPU kernel, reducing launch overhead and increasing computational density.

Accurate TPS measurement requires controlled, reproducible conditions to establish a valid Performance Baseline.

• Steady-State Measurement: TPS should be measured after the Cold Start Latency period, when caches are warm and performance is stable.
• Defined Workload: Use either a Synthetic Workload (e.g., fixed prompt/output length distribution) or a representative Real-World Workload. The workload defines the Average Tokens Per Request.
• Industry Benchmarks: MLPerf Inference provides standardized benchmarks for comparing TPS across different systems and software stacks under fair conditions.
• Load Testing: Gradually increasing concurrent request load while measuring TPS and latency reveals the system's Throughput-Latency Curve and saturation point.

TPS is specific to token-generating models. It must be distinguished from related throughput metrics:

• Queries Per Second (QPS): Measures complete requests per second. QPS = TPS / (Average Output Tokens per Request). QPS is useful for cost-per-request analysis but obscures the computational cost of long generations.
• Inferences Per Second (IPS): A general metric for non-LLM models (e.g., classification). For LLMs, one 'inference' can generate many tokens, making IPS less precise than TPS.
• Context: TPS measures output generation speed. The processing of input prompts (affecting Time to First Token) is a separate cost. High TPS systems must also optimize prompt processing to maintain low TTFT under concurrency.

Latency is the time delay between a request being submitted and the response being received, measured in milliseconds. It is the primary user-facing metric for responsiveness. For language models, latency is often broken down into two key components:

• Time to First Token (TTFT): The initial delay before any output begins.
• Time per Output Token (TPOT): The incremental delay for each subsequent token. High TPS does not guarantee low latency; a system can batch many requests to achieve high throughput while individual user requests experience significant delay.

Throughput is the rate at which a system processes work. While TPS measures token generation speed, other common throughput metrics provide different perspectives on system capacity:

• Queries Per Second (QPS): Measures complete requests processed per second, agnostic to output length.
• Inferences Per Second (IPS): Similar to QPS, used for non-generative tasks like classification. The relationship between throughput (QPS/TPS) and latency is non-linear and is visualized using a throughput-latency curve. As load increases, latency typically rises sharply once the system reaches its saturation point.

Time to First Token is the latency from request submission until the first output token is generated. This metric captures the initial computational overhead, which includes:

• Prompt encoding and processing through the full model.
• KV Cache initialization for the attention mechanism.
• Any prefill computation in the transformer architecture. TTFT is highly sensitive to prompt length and is a critical metric for interactive applications like chatbots, where users perceive immediate feedback. Optimizations like continuous batching can negatively impact TTFT for individual requests if the batch isn't full.

Time per Output Token is the average latency to generate each subsequent token after the first. This represents the incremental, autoregressive generation cost. TPOT is primarily determined by:

• The speed of reading from the KV Cache and performing attention calculations.
• The efficiency of the model's decoding step.
• Memory bandwidth constraints, making it often memory-bound. TPOT multiplied by the number of output tokens defines the total generation time after TTFT. Techniques like speculative decoding aim to reduce effective TPOT by having a smaller draft model propose multiple tokens for verification by the main model in a single step.

Tail latency refers to the high-percentile latencies (e.g., P90, P95, P99) in a request distribution. While average latency might be low, a small percentage of requests can be significantly slower, degrading user experience. Common causes in inference systems include:

• Resource contention from concurrent requests.
• Garbage collection pauses or other system-level interference.
• Variability in request complexity (e.g., longer prompts). Managing tail latency is crucial for Service Level Objectives (SLOs). Performance is often measured against P99 latency targets to ensure consistency for most users.

Queries Per Second is a system-level throughput metric measuring the number of complete inference requests processed per second. Unlike TPS, QPS treats each request as a single unit, regardless of its input or output token count. This makes QPS useful for:

• Capacity planning and scaling infrastructure.
• Comparing systems where request sizes are similar.
• Understanding overall system load. The relationship between QPS and TPS is: TPS ≈ QPS * (Average Output Tokens per Request). For variable-length requests, TPS and QPS can tell different stories about system performance and efficiency.