Time to First Token (TTFT)

The computational graph of the model is the primary determinant of TTFT. Larger models with more parameters and layers require more sequential operations to process the prompt.

• Parameter Count: Models with billions of parameters (e.g., 70B, 180B) have vastly more weights to load and compute with than smaller models (e.g., 7B).
• Context Window: Processing a long context (e.g., 128K tokens) involves computing attention over all prompt tokens, which scales quadratically with sequence length in standard attention, significantly increasing TTFT.
• Architecture Choices: Models using Grouped-Query Attention (GQA) or Multi-Query Attention (MQA) reduce the memory footprint of the KV Cache during the prefill phase compared to standard Multi-Head Attention, potentially lowering TTFT.

TTFT is heavily dependent on the raw FLOP/s (floating-point operations per second) and memory bandwidth of the hardware, and how effectively computation is parallelized.

• GPU/Accelerator Type: An H100 or A100 GPU provides significantly higher compute throughput and memory bandwidth than a T4 or consumer-grade GPU, directly reducing TTFT.
• Tensor Parallelism: Splitting the model across multiple GPUs (model parallelism) allows layers to be computed in parallel, drastically cutting down the sequential computation time for the initial forward pass.
• Kernel Optimization: Highly optimized, fused CUDA kernels for matrix multiplications and attention operations provided by frameworks like FlashAttention-2 minimize overhead and maximize hardware utilization.

The prefill phase is the compute-intensive, parallelizable step where the entire input prompt is processed to generate the KV Cache. This phase dominates TTFT for non-trivial prompts.

• Prompt Length: TTFT scales approximately linearly with the number of tokens in the input prompt during this phase. A 500-token prompt takes roughly 5x longer to prefill than a 100-token prompt.
• Arithmetic Intensity: The prefill phase involves large matrix multiplications and is typically compute-bound on modern accelerators, meaning performance is limited by FLOP/s, not memory bandwidth.
• Continuous Batching Impact: In a continuous batching system, a new request must wait for the current prefill batch to complete if the system is optimized for throughput, which can increase its TTFT.

The time to load model weights and intermediate activations from various levels of memory is a key factor, especially for large models.

• Model Loading (Cold Start): The initial load of model weights from disk (e.g., NVMe SSD) into GPU HBM (High-Bandwidth Memory) incurs a significant one-time penalty, affecting the first request's TTFT.
• Weight Offloading: Systems using CPU RAM offloading or NVMe offloading to host larger models than GPU memory can hold suffer from severe TTFT penalties due to slow PCIe or disk I/O during computation.
• KV Cache Initialization: Allocating and writing the initial KV Cache for the full prompt context in HBM is a memory-bound operation that contributes to TTFT.

The inference server software and its scheduling policies create overhead that adds to TTFT.

• Framework Overhead: Python-based servers with deep framework stacks (e.g., Hugging Face transformers) have higher overhead than optimized, low-level C++ runtimes (e.g., vLLM, TensorRT-LLM, SGLang).
• Request Scheduling: First-Come, First-Served (FCFS) scheduling in a busy system can queue a request behind others, increasing its TTFT. More advanced schedulers may prioritize short prompts.
• Dynamic Batching: Waiting to group multiple requests into a batch for more efficient prefill (dynamic batching) introduces scheduling delay, trading off individual TTFT for higher aggregate throughput.

The broader runtime environment and shared resource usage significantly impact tail latency metrics like P99 TTFT.

• Multi-Tenancy & Resource Contention: On a shared GPU cluster, other jobs or inference requests compete for SMs (Streaming Multiprocessors), HBM bandwidth, and PCIe bandwidth, causing unpredictable increases in TTFT.
• Host CPU Bottlenecks: Preprocessing (tokenization), scheduling logic, and data movement orchestration on the host CPU can become a bottleneck, especially for high request rates.
• Network Overhead: For client-server deployments, the network round-trip time (RTT) is added to the user-perceived TTFT, though it is separate from the model's computational TTFT.

Time per Output Token is the average latency for generating each subsequent token after the first in an autoregressive language model. It represents the incremental, per-token generation cost during the decoding phase.

• Key Difference from TTFT: While TTFT measures the initial prompt processing overhead, TPOT measures the sustained generation speed.
• System Bottleneck: A high TPOT often indicates a memory-bound workload, constrained by the speed of reading the KV Cache from GPU memory for each decoding step.
• Impact on User Experience: Directly affects the perceived "typing" speed of a streaming model response.

Tail latency refers to the high-percentile latencies (e.g., P95, P99) in a request distribution, representing the slowest requests that often determine worst-case user-perceived performance.

• Relation to TTFT: A system may have a good average TTFT but suffer from high tail latency, causing inconsistent user experience. This is critical for Service Level Objectives (SLOs).
• Common Causes: Resource contention from concurrent requests, garbage collection pauses, host-level interference, or variability in prompt length and complexity.
• Measurement: Essential for evaluating production readiness, as it indicates system stability under load.

Cold start latency is the additional delay incurred when initializing an inference service from an idle state, including model loading, runtime compilation, and cache warm-up before reaching steady-state performance.

• Distinction from TTFT: TTFT is measured during steady-state operation. Cold start includes TTFT plus the one-time initialization overhead.
• Critical in Serverless: A major challenge for serverless ML deployments where instances scale to zero. Techniques like pre-warmed pools are used to mitigate it.
• Components: Includes loading model weights into GPU memory, compiling kernels (e.g., with TensorRT or Triton), and initializing system caches.

Throughput measures the rate of work completed by an inference system. For language models, it's commonly measured in Tokens Per Second (TPS) or Queries Per Second (QPS).

• Trade-off with TTFT: Optimizing for maximum throughput (e.g., using large batch sizes) often increases TTFT, as requests wait in a queue for continuous batching. Optimizing for low TTFT (small batches, high priority) can reduce total throughput.
• System Capacity: Defined by the saturation point on the throughput-latency curve.
• Measurement: TPS is the total output tokens generated per second across all requests, providing a holistic measure of system efficiency.

Continuous batching (or iterative batching) is a dynamic scheduling technique that groups multiple inference requests into a single computational batch and updates the batch composition in real-time as requests finish, maximizing GPU utilization.

• Direct Impact on TTFT: Efficient batching reduces queueing delay for incoming requests, improving TTFT under load. However, it must balance new requests with ongoing generations.
• Core Mechanism: As some sequences in a batch finish generation, their slots are immediately filled with new requests, keeping the GPU constantly occupied.
• Implementation: A key feature in high-performance inference servers like vLLM and NVIDIA Triton.

KV Cache management involves the efficient storage and retrieval of key-value pairs from the transformer's attention mechanism during autoregressive decoding, which is the primary state maintained between token generation steps.

• Primary Influence on TTFT: The initial computation and population of the KV Cache for the prompt tokens is a major component of TTFT. Efficient prompt encoding is crucial.
• Memory Bound: TPOT is largely determined by the bandwidth of reading this cache. Techniques like paged attention (vLLM) optimize memory usage and reduce fragmentation.
• Optimization: Critical for supporting long contexts, as the KV Cache size grows linearly with sequence length.