Tokens Per Second (TPS)

The structural complexity of the model directly dictates computational cost per token. Mixture of Experts (MoE) architectures achieve higher TPS by activating only a fraction of total parameters per token, while dense models scale compute linearly with size. Grouped-Query Attention (GQA) reduces KV cache bandwidth, accelerating generation compared to full Multi-Head Attention. Smaller parameter counts generally yield higher TPS, but quantization can make larger models viable on fixed hardware.

Reducing numerical precision from 16-bit to 4-bit integers using formats like GGUF dramatically decreases memory bandwidth pressure and compute requirements. This allows larger models to fit entirely in GPU VRAM or even run on CPU with acceptable TPS. QLoRA enables fine-tuning at this precision. The trade-off is a minor perplexity increase for a substantial throughput gain, making quantization the single most impactful software lever for TPS on consumer or edge hardware.

The serving software stack is critical. vLLM utilizes PagedAttention to manage KV cache memory with near-zero fragmentation, enabling high Continuous Batching efficiency. Llama.cpp is heavily optimized for CPU and hybrid GPU inference on personal devices. NVIDIA Triton Inference Server provides dynamic batching and model ensembles for enterprise deployments. The choice of engine and its ability to overlap computation with memory transfers directly governs achievable TPS under concurrent load.

For token generation, performance is often memory-bound rather than compute-bound. The rate at which model weights can be streamed from VRAM or system RAM to the processing units is the primary bottleneck. High-bandwidth memory (HBM) on data-center GPUs provides a massive advantage. On consumer hardware, ensuring the model fits entirely in fast GPU memory avoids the catastrophic TPS drop caused by offloading layers to slower system RAM or swap.

The KV Cache stores attention keys and values to avoid recomputation for each new token. Its size grows linearly with batch size and sequence length. Inefficient memory allocation leads to fragmentation and waste, capping the number of concurrent sequences. PagedAttention solves this by allocating cache in blocks, while Prefix Caching reuses the cache for identical prompt preambles. Effective cache management directly increases the maximum batch size and thus total system TPS.

TPS is measured on output tokens, but the prompt's length impacts the prefill phase and initial memory allocation. A very long prompt with a short generation will show a lower average TPS if prefill time is included. Speculative Decoding uses a smaller draft model to propose tokens rapidly, which the large model verifies in parallel, effectively decoupling TPS from the large model's inherent latency for low-batch, interactive scenarios.

The latency metric measuring the delay between submitting a prompt and receiving the first output token. While TPS measures generation speed after the first token, TTFT captures the prefill phase where the model processes the entire input prompt. A high TTFT can make a system feel sluggish even if TPS is high. Optimizing TTFT often involves prefix caching and efficient attention kernels like Flash Attention.

A serving technique that maximizes GPU utilization by dynamically adding new sequences to a running batch without waiting for all sequences to complete. This directly increases aggregate TPS by preventing idle compute cycles. Unlike static batching, continuous batching allows high-priority requests to jump the queue, improving both throughput and latency under load. Implemented in engines like vLLM and NVIDIA Triton.

A compression technique that reduces numerical precision of weights from FP16 to INT8 or INT4. Quantization decreases memory bandwidth pressure, allowing more data to move through the GPU per second. This often yields a 2-4x increase in TPS on the same hardware, though aggressive quantization may degrade output quality. Formats like GGUF are specifically designed for high-throughput quantized inference.

An acceleration method that uses a small draft model to generate multiple candidate tokens quickly, which a larger target model then verifies in parallel. This can increase TPS by 2-3x without any loss in output quality, as the target model guarantees correctness. It trades a small amount of additional compute for dramatically faster wall-clock generation.

A distributed strategy that splits individual weight matrices across multiple GPUs to serve models too large for a single device. While it enables running massive models, it introduces inter-GPU communication overhead that can reduce per-device TPS. Efficient tensor parallelism requires high-bandwidth interconnects like NVLink to minimize this communication bottleneck.