Tokens Per Second (TPS)

The fundamental design of the language model is the primary determinant of TPS.

• Parameter Count: Larger models (e.g., 70B+ parameters) require more computations per token, reducing TPS compared to smaller models (e.g., 7B parameters) on identical hardware.
• Architecture Family: Architectures like Mixture of Experts (MoE) can achieve higher TPS than dense models of equivalent parameter count by activating only a subset of weights per token.
• Context Window Length: Processing very long sequences increases memory bandwidth pressure for the Key-Value (KV) Cache, which can bottleneck TPS.

The physical infrastructure executing the model defines the upper bound for TPS.

• GPU/Accelerator Type: Modern AI accelerators (e.g., NVIDIA H100, Google TPU v5e) with specialized tensor cores provide vastly higher TPS than general-purpose CPUs.
• Memory Bandwidth: The speed at which model weights can be read from GPU VRAM (High Bandwidth Memory) is often the limiting factor for TPS, a phenomenon known as being memory-bound.
• Batching Strategy: Continuous batching dynamically groups requests, maximizing GPU utilization and significantly improving aggregate TPS over static batching.

Software-level optimizations are essential for achieving peak hardware performance.

• Quantization: Reducing model weight precision from 16-bit (FP16) to 8-bit (INT8) or 4-bit (NF4) halves or quarters memory bandwidth requirements, directly boosting TPS with minimal accuracy loss.
• Kernel Optimization: Using highly optimized, low-level compute kernels (e.g., via FlashAttention) reduces the operational overhead of attention mechanisms.
• Compiler Techniques: Frameworks like TensorRT-LLM or vLLM apply graph optimizations, kernel fusion, and expert memory management to maximize TPS.

The surrounding serving infrastructure introduces latency that impacts effective TPS.

• Pre/Post-Processing: Tokenization, detokenization, and output formatting add fixed latency per request, reducing overall system TPS.
• Network Latency: In distributed systems, communication between orchestrators, model servers, and tokenizers adds delay.
• Queuing & Scheduling: Under high load, request queuing and scheduler overhead can become the bottleneck, capping realized TPS below the hardware's theoretical maximum.

The algorithm used to generate tokens directly controls the number of serial inference steps required.

• Greedy Decoding: Produces one token per forward pass. It is the fastest (highest TPS) but can lead to repetitive or low-quality output.
• Sampling (Top-k, Top-p): Introduces randomness by sampling from the probability distribution, maintaining quality with a negligible TPS impact versus greedy decoding.
• Beam Search: Explores multiple sequence possibilities in parallel, requiring k times more computations (where k is the beam width), drastically reducing TPS.

While TPS measures steady-state throughput, Time to First Token (TTFT) measures the latency to start streaming. They are often in tension.

• High TPS, High TTFT: Systems optimized for large batch processing may have high TPS but also high TTFT as they wait to fill a batch.
• Low TPS, Low TTFT: Systems prioritizing responsiveness may process requests immediately (low TTFT) but sacrifice aggregate TPS due to poor GPU utilization.
• Optimization Goal: The ideal system balances both, using techniques like continuous batching and prefill-decoding separation to minimize TTFT while maximizing TPS.

Time to First Token measures the latency from sending a request to a generative model until the client receives the first output token. This is a critical user-perceived latency metric, distinct from overall throughput.

• Primary Driver: Initial model processing and prompt context loading.
• Impact: Directly affects user experience for interactive applications. High TTFT makes agents feel unresponsive.
• Relationship to TPS: A system can have high TPS (good steady-state throughput) but suffer from high TTFT if there are long initialization or queuing delays.

End-to-End Latency is the total duration of a complete user interaction with an AI agent, from initial input to final, actionable output. It encompasses all processing stages.

• Components: Includes TTFT, inter-token latency (1/TPS), agent reasoning time, external tool call execution, and network overhead.
• Holistic View: The ultimate measure of agent responsiveness from a user's perspective.
• Optimization Focus: Requires profiling the entire pipeline, as improving just TPS may not significantly reduce end-to-end time if other stages (e.g., planning, API calls) are the bottleneck.

Throughput, often measured in Requests Per Second (RPS), is the rate at which a system completes entire requests. For AI agents, this is a higher-level metric than TPS.

• System-Level Metric: Measures completed agent tasks per second, where a task may involve multiple LLM calls and tool executions.
• Concurrency Dependency: Maximum RPS is achieved by processing many requests simultaneously via techniques like continuous batching.
• Trade-off with Latency: Increasing RPS (handling more concurrent requests) often increases per-request latency (P50, P99) due to resource contention.

Cost Per Thousand Tokens (CPT) is the standardized pricing metric for cloud LLM inference. Performance optimization must balance speed (TPS) with cost efficiency.

• Direct Link to TPS: Higher TPS can reduce cost-per-request by completing work faster, but aggressive optimization for TPS (e.g., larger batch sizes) may increase cost-per-token if it reduces hardware utilization efficiency.
• Telemetry Integration: Agent cost telemetry pipelines track token consumption across sessions to attribute expenses and calculate return on investment.
• Engineering Decision: Choosing a model or optimization technique involves evaluating the TPS/CPT trade-off for a given latency SLO.

Concurrency Level is the number of simultaneous requests or user sessions a system is processing. It is a key load parameter that directly impacts observed TPS and latency.

• Dynamic Scaling: Systems auto-scale to maintain TPS/RPS as concurrency increases, up to a saturation point.
• Performance Testing: Load tests ramp up concurrency to find the point where latency degrades or errors rise, defining system limits.
• Batching Efficiency: Higher concurrency enables more efficient continuous batching in the inference engine, improving aggregate TPS but potentially increasing individual request latency.

A Service Level Objective is a target for a reliability or performance metric, such as "P99 latency < 2 seconds" or "TPS > 1000." SLOs define the expected experience for agent systems.

• SLO Composition: Often includes a latency target (e.g., end-to-end or TTFT) and a throughput target (TPS/RPS).
• Error Budget: The allowable violation of an SLO, used to govern release risk. A drop in TPS below target consumes the error budget.
• Observability Requirement: Meeting SLOs requires comprehensive agent telemetry pipelines to measure TPS, latency, and errors in real-time.