Tokens per Second (TPS)

The foundational layer determining peak TPS. Key components include:

• GPU/Accelerator: The type (e.g., H100, A100), memory bandwidth, and number of devices. Higher FLOPs and memory bandwidth directly increase token generation speed.
• Memory Capacity: The size of VRAM dictates the maximum model size and batch size that can be loaded. Insufficient memory forces costly swapping to CPU RAM, crippling TPS.
• Interconnect: The speed of links between GPUs (e.g., NVLink, InfiniBand) is crucial for multi-GPU inference, affecting how efficiently workloads are distributed.

The model's own design imposes fundamental constraints.

• Parameter Count: Larger models (e.g., 70B vs 7B parameters) require more computations per token, reducing TPS for a given hardware setup.
• Attention Mechanism: The complexity of the attention operation (e.g., standard vs. grouped-query or multi-query attention) significantly impacts the computational graph. Optimized variants can dramatically improve TPS.
• Precision: Using lower numerical precision (e.g., FP16, BF16, INT8, or INT4 quantization) reduces memory footprint and increases compute efficiency, often doubling or tripling TPS compared to FP32.

Software orchestration of requests is paramount for achieving high aggregate TPS.

• Static vs. Continuous Batching: Static batching processes a fixed set of requests together. Continuous batching (or iteration-level batching) dynamically adds new requests to the batch as others finish, dramatically improving GPU utilization and overall system TPS.
• KV Cache Management: Efficient caching of previous tokens' Key and Value vectors in the attention mechanism avoids redundant computation. The size and management strategy of this cache directly affect memory usage and decode speed.
• Serving Framework: Specialized frameworks like vLLM, TensorRT-LLM, or TGI implement these optimizations, offering vastly higher TPS than naive model serving.

The nature of the user's input and desired output determines the workload for each request.

• Input (Prompt) Length: Longer prompts increase the time and memory required for the initial prefill phase, impacting Time to First Token (TTFT) and the efficiency of the initial batch computation.
• Output (Generation) Length: The number of tokens to be generated defines the duration of the decode phase. Longer generations increase total request latency but, with efficient batching, can improve overall system TPS by keeping GPUs saturated.
• Decoding Strategy: Greedy decoding (selecting the highest probability token) is fastest. Sampling methods (top-k, top-p, temperature) introduce minor overhead. Complex strategies like beam search multiply the computational cost, severely reducing TPS.

Infrastructure factors outside the core model execution can become bottlenecks.

• Host CPU & I/O: The CPU must be powerful enough to handle tokenization, detokenization, and framework overhead without stalling the GPU. Slow disk I/O can delay model loading.
• Network Latency: In distributed or client-server setups, the time to transmit prompts and stream tokens adds to total response time, though it doesn't reduce the model's intrinsic TPS.
• Orchestration & Queuing: Load balancers, API gateways, and request queues add minimal but non-zero latency. Under high load, queuing delays can dominate the user-perceived latency, masking the actual TPS of the inference engine.

TPS is a variable metric, and its reported value must be interpreted with context.

• Peak vs. Sustained TPS: Peak TPS is measured under ideal, saturated conditions (large batch, optimal input/output lengths). Sustained TPS reflects performance under a realistic, variable production workload and is typically lower.
• Benchmark Conditions: Always ask: Batch size? Input/Output length? Hardware spec? Model precision? Without this context, a TPS number is meaningless for comparison.
• Trade-off with Latency: Maximizing TPS often involves large batch sizes, which can increase latency for individual requests (higher TTFT, queue time). Engineering must balance throughput with latency Service Level Objectives (SLOs).

Continuous batching dynamically groups incoming inference requests into a single computational batch, adding new requests as others finish generation. This maximizes GPU utilization by eliminating idle time between static batches.

• Key Benefit: Dramatically improves hardware throughput compared to static batching.
• Implementation: Used in serving frameworks like vLLM and TGI (Text Generation Inference).
• Impact: Can increase effective TPS by 5-10x for workloads with variable request lengths.

The Key-Value (KV) Cache stores computed attention key and value vectors for previously generated tokens, preventing redundant computation during autoregressive decoding. Optimizing its management is critical for TPS.

• Memory Bottleneck: KV cache size grows linearly with batch size and sequence length, often becoming the limiting factor.
• Techniques: Include paged attention (vLLM) to eliminate memory fragmentation and quantized KV caches (e.g., FP8) to reduce memory bandwidth pressure.
• Result: Efficient KV cache management directly reduces decode latency, boosting TPS.

Quantization reduces the numerical precision of a model's weights and activations (e.g., from 16-bit to 8-bit or 4-bit). This decreases memory footprint and increases compute efficiency.

• Types: Post-Training Quantization (PTQ) applies compression after training; Quantization-Aware Training (QAT) fine-tunes the model to compensate for precision loss.
• Frameworks: GPTQ, AWQ, and bitsandbytes are common libraries.
• TPS Gain: Can enable 2-4x higher batch sizes on the same hardware, proportionally increasing TPS.

Flash Attention is an optimized algorithm that computes exact attention with significantly fewer memory reads/writes (I/O operations) by fusing computation kernels. This reduces the time spent on the attention mechanism, which is often the computational bottleneck.

• Principle: Leverages hardware memory hierarchy (SRAM vs. HBM) for efficient data movement.
• Impact: Reduces prefill latency (improving TTFT) and increases overall compute throughput, contributing to higher TPS, especially for long contexts.
• Extension: FlashAttention-2 and related kernel fusions for other operations (e.g., MLP layers) provide further gains.

Speculative decoding uses a small, fast draft model to predict a sequence of several future tokens. These are then verified in a single forward pass by the larger target model, rejecting mismatches.

• Mechanism: Amortizes the cost of the large model's forward pass over multiple tokens.
• Requirement: The draft model must have high predictive accuracy (e.g., a distilled version of the target).
• Outcome: Can achieve 2-3x latency reduction for the same model, effectively doubling or tripling TPS in supported scenarios.

Maximizing TPS requires tailoring the inference stack to the underlying accelerator architecture (GPU, NPU, etc.).

• Kernel Optimization: Writing custom CUDA/HIP kernels or using compiler frameworks like OpenAI Triton for optimal hardware utilization.
• Model Compilation: Using NVIDIA TensorRT or OpenXLA to compile the model graph into a single, optimized executable kernel.
• Memory Allocation: Employing efficient, pre-allocated memory pools to reduce overhead from dynamic allocation during inference.
• Result: These low-level optimizations provide the final performance layer, often yielding 20-50% TPS improvements over generic implementations.

Time to First Token measures the latency from request submission to the arrival of the first output token. This metric is dominated by the prefill stage, where the model processes the entire input prompt to compute initial logits and populate the KV Cache. High TTFT can indicate insufficient compute resources for prompt processing or inefficient batching.

• Primary Driver: Computational cost of the attention mechanism over the full input sequence.
• Key Trade-off: Often inversely related to TPS; optimizing for one can negatively impact the other.
• User Impact: Directly affects perceived responsiveness in streaming applications.

Inter-Token Latency is the average time interval between the generation of consecutive output tokens during the autoregressive decode stage. It is the reciprocal of Tokens per Second (TPS = 1 / Inter-Token Latency). This metric determines the fluency of a streaming response.

• Primary Driver: Memory bandwidth for reading the KV Cache and compute for generating the next token.
• Optimization Target: Techniques like continuous batching and optimized attention kernels aim to minimize this latency.
• Monitoring Context: Should be tracked via latency percentiles (P50, P90, P99) to understand tail performance and variability.

Continuous Batching is an inference optimization technique that dynamically adds new requests to a running batch as previous requests finish generation, instead of waiting for the entire batch to complete. This dramatically improves GPU utilization and overall system throughput (TPS).

• Mechanism: The scheduler manages a pool of active requests, adding and removing them from the computational batch in real-time.
• Impact on TPS: Can increase effective TPS by 5-10x compared to static batching by eliminating idle GPU cycles.
• Consideration: Increases implementation complexity for the inference server to manage variable-length sequences and partial completions.

The Key-Value (KV) Cache is a critical memory structure in transformer-based LLM inference. It stores the computed key and value vectors for all previous tokens in a sequence, allowing the attention mechanism to reference past context without recalculating it for each new token.

• Purpose: Eliminates redundant computation during autoregressive decoding, drastically speeding up token generation.
• Trade-off: Consumes significant GPU memory (often multiple GBs per request), which can limit batch size and throughput.
• Optimization: Techniques like paged attention and quantization of the KV cache are used to manage memory footprint and improve TPS.

Service Level Indicators (SLIs) are quantitative measures of an LLM service's performance, such as average TPS, P99 latency, or error rate. A Service Level Objective (SLO) is a target value for an SLI that defines acceptable service performance.

• Example SLO: "99% of requests shall have a Time to First Token below 500ms and a sustained TPS above 45 tokens/second."
• Error Budget: The allowable amount of SLO violation over a period, used to govern release velocity and prioritization of reliability work.
• Foundation: Essential for defining and monitoring production-grade LLM performance beyond isolated TPS measurements.

Latency Percentiles describe the distribution of response times. The P50 (median), P90, and P99 latencies represent the maximum latency experienced by 50%, 90%, and 99% of requests, respectively. They are crucial for understanding tail latency.

• Importance for TPS: Average TPS can mask poor user experience. A high P99 latency for Inter-Token Latency means 1% of users experience very slow streaming, even if average TPS is good.
• Monitoring: Requires high-cardinality metric collection to segment by model, hardware, and request parameters.
• Use Case: Informing capacity planning and autoscaling decisions to meet SLOs under variable load.