Tokens Per Second (TPS)

The physical accelerator is the primary determinant of peak TPS. Key specifications include:

• Compute Cores (CUDA/Stream Processors): Dictate raw parallel processing power for matrix multiplications.
• Memory Bandwidth (GB/s): The speed at which model weights and the KV cache can be read into on-chip memory; often the bottleneck for large models.
• VRAM Capacity: Limits the maximum model size and batch size that can be loaded. High TPS requires keeping the computational units saturated, which is constrained by how quickly data can be fed to them.

Example: An NVIDIA H100 with 1,979 GB/s memory bandwidth and dedicated Transformer Engine cores will achieve significantly higher TPS than an A100 (1,555 GB/s) for the same model.

The neural network's design imposes fundamental computational constraints.

• Parameter Count: Larger models (e.g., 70B vs. 7B) have more weights to load and compute, reducing TPS for a given hardware setup.
• Attention Mechanism: Standard quadratic-complexity attention is more demanding than efficient variants like FlashAttention or grouped-query attention (GQA).
• Context Window Length: Longer sequences increase the size of the KV Cache, consuming more memory bandwidth and computation during decoding.

The relationship is inverse: TPS ∝ 1 / (Model Size × Sequence Length).

Software-level techniques that maximize hardware utilization.

• Continuous Batching: Groups multiple user requests (of varying lengths) into a single batch, dynamically updating it as requests finish, maximizing GPU utilization.
• PagedAttention (vLLM): Manages the KV Cache in non-contiguous memory blocks, drastically reducing fragmentation and waste, allowing for larger effective batch sizes.
• Model Quantization: Using INT8 or GPTQ/AWQ (4-bit) reduces the memory footprint and bandwidth requirements of weights, directly increasing potential TPS.
• Speculative Decoding: Uses a small draft model to propose token sequences, verified in parallel by the main model, reducing the number of costly large model runs.

How users interact with the system defines the achievable TPS.

• Batch Size: The number of requests processed simultaneously. Larger batches improve GPU utilization but increase time to first token (TTFT).
• Output Token Length: Generating longer responses increases the total computational work per request, reducing the overall request throughput.
• Traffic Concurrency & Distribution: Steady, high-concurrency traffic allows for optimal batching. Sparse, bursty traffic leads to poor GPU saturation and lower effective TPS.
• Prompt Complexity: Longer, more complex input prompts increase the computational cost of the initial prefill phase before token generation begins.

Infrastructure inefficiencies that cap realized TPS.

• Cold Starts: The latency and throughput penalty when a new server instance must load a model into GPU memory before serving requests.
• Host-to-Device Data Transfer: Time spent moving prompts from CPU/RAM to GPU VRAM.
• Inter-Node Communication: In tensor or pipeline parallel deployments, network latency between GPUs can become a bottleneck.
• Scheduling & Queueing Delays: Overhead from the serving framework (e.g., TorchServe, Triton) and load balancers in routing and preparing requests.

Maximizing TPS is not an isolated goal; it exists within a critical trilemma with cost and latency.

• High TPS via Large Batches: Increases throughput and lowers cost per token but raises Tail Latency (P99) for individual users.
• Low Latency Prioritization: Requires smaller batches or even no batching, maximizing responsiveness for a single user but drastically reducing overall TPS and increasing cost.
• Optimization Target: Engineering teams must configure their serving systems (batch size, autoscaling rules) based on the Service Level Objective (SLO)—whether it's minimizing cost, maximizing throughput, or guaranteeing low-latency user experiences.

Reducing the numerical precision of model weights and activations decreases memory bandwidth requirements and accelerates computation.

• INT8/FP8 Quantization: Using 8-bit integers or floats instead of 16-bit (FP16) or 32-bit (FP32) can double theoretical TPS by enabling faster matrix operations on supported hardware.
• GPTQ & AWQ: Advanced 4-bit post-training quantization methods that minimize accuracy loss. A 4-bit quantized model requires 1/4 of the original GPU memory, allowing larger batch sizes or the use of smaller, cheaper GPUs for the same model, directly boosting TPS per dollar. Quantization is often the first step in any TPS optimization pipeline.

Matching the model architecture to the underlying hardware is essential for peak TPS.

• Tensor Parallelism: Splits individual model layers across multiple GPUs. Essential for serving models larger than a single GPU's memory (e.g., 70B+ parameter models). While it introduces communication overhead, it's the only way to serve such models at all.
• Optimized Kernels: Using framework-specific optimized kernels (e.g., FlashAttention for faster attention computation, fused operators) reduces core computational latency.
• Right-Sizing Instances: Selecting the optimal cloud GPU instance (e.g., A100 80GB vs. H100) based on model size, batch size, and latency targets to maximize TPS per dollar.

TPS is a system-level metric, not just a model-level one. Infrastructure decisions have a major impact.

• Autoscaling: Dynamically adding or removing model inference pods based on request queue length to handle traffic spikes without over-provisioning.
• Load Shedding & Rate Limiting: Proactively rejecting low-priority requests during overload to protect the system's ability to serve high-priority requests at target TPS and latency (P99).
• Optimized Cold Starts: Using techniques like pre-warming instances or snapshotting loaded models to minimize the latency penalty when scaling out, ensuring consistent TPS for all users.

Cost per token is the fundamental unit economics metric for LLM inference, calculated by dividing the total expense of a generation task by the number of tokens processed. It is directly influenced by TPS: a higher TPS for a given hardware setup lowers the cost per token by amortizing fixed compute costs over more output. This metric is essential for forecasting budgets and comparing the efficiency of different model serving configurations.

• Primary Drivers: Model size, quantization level, cloud instance cost, and achieved TPS.
• Calculation: (Instance Cost per Hour / Tokens Generated per Hour).
• Use Case: A CTO uses cost per token to decide between a faster, more expensive instance (high TPS, potentially lower cost/token at high load) and a slower, cheaper one.

Tail latency, measured at the 95th or 99th percentile (P95/P99), represents the worst-case response times experienced by a small fraction of user requests. It has an inverse relationship with TPS in a loaded system. Aggressive optimization for maximum average TPS (e.g., through very large batch sizes) can increase queueing delays and variability, degrading tail latency. Engineering for consistent user experience requires balancing high TPS with predictable, low tail latency.

• Trade-off: Maximizing TPS can sometimes increase latency variance.
• Monitoring: Critical for SLA compliance in user-facing applications.
• Impact: A chatbot with high TPS but poor P99 latency will feel sluggish to some users, harming perceived performance.

Continuous batching (or in-flight batching) is an advanced inference optimization technique that dramatically improves TPS. Instead of waiting for a full batch of requests to finish, it dynamically inserts new requests into a running batch as soon as GPU resources free up from completed sequences. This eliminates the idle time present in static batching, leading to significantly higher GPU utilization and token throughput, especially for workloads with variable-length prompts and generations.

The KV (Key-Value) Cache is a memory optimization for transformer decoder blocks that is fundamental to achieving high TPS during autoregressive generation. It stores computed key and value tensors for previously processed tokens, preventing their recalculation on each new generation step. Efficient management of the KV cache is critical: its memory footprint grows with batch size and sequence length, directly limiting concurrent requests and thus potential TPS. Techniques like PagedAttention (used in vLLM) optimize this memory usage.

vLLM is a high-throughput LLM serving engine that employs the PagedAttention algorithm to optimize memory management for the KV Cache. PagedAttention treats the KV cache as non-contiguous, paged blocks in memory—similar to virtual memory in an operating system. This drastically reduces memory fragmentation and waste, allowing for much larger effective batch sizes and higher concurrency. The result is a significant increase in achieved TPS compared to naive serving systems, especially for long sequences.

Model quantization is a compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 16-bit to 8-bit or 4-bit integers). This directly boosts TPS by:

• Reducing memory bandwidth pressure, allowing faster weight loading.
• Enabling the use of specialized integer compute units on modern GPUs.
• Allowing larger batch sizes to fit in the same GPU memory.

Methods like GPTQ (post-training) and AWQ (activation-aware) are used to apply quantization with minimal accuracy loss, making high TPS serving on cost-effective hardware feasible.