Time to First Token (TTFT)

The time an inference request spends waiting in a scheduler's queue before execution begins. This is a major, often variable, component of TTFT under load.

• Primary Driver: High concurrent requests relative to system capacity.
• Impact: Can dominate TTFT during traffic spikes, even if compute is fast.
• Mitigation: Effective load balancing, request prioritization, and autoscaling to reduce queue depth.

The additional delay incurred when the first request(s) target a model instance that is not loaded in memory. This includes:

• Model Loading: Reading weights from disk/network into GPU memory.
• Initialization: One-time setup of runtime contexts and model execution graphs.
• Cache Warming: Initial forward passes to populate caches.
• Impact: Critical for serverless or infrequently used endpoints, causing sporadic high TTFT.

The time required for the model to process the static input prompt and context through a full forward pass. This phase must complete before any token generation can begin.

• Mechanics: The entire input sequence is processed in parallel to generate the initial Key-Value (KV) cache for the attention layers.
• Bottlenecks: Scales linearly with prompt length and is computationally intensive. GPU kernel launch overhead and memory bandwidth can be limiting factors.
• Optimization: Techniques like operator fusion and efficient model execution graph compilation (e.g., with TensorRT) are applied here.

The latency of the first autoregressive decoding step, where the model generates the initial output token conditioned on the KV cache from the prefilling phase.

• Process: A single, often memory-bound, sampling operation from the model's output logits.
• Contrast with TPOT: This step is typically slower than subsequent Time Per Output Token (TPOT) steps due to fixed overheads that are amortized over later tokens.
• Influence of System: Efficient management of the KV cache (e.g., via PagedAttention in vLLM) is crucial to minimize this step's latency.

The latency contributed by data transfer and protocol processing outside the core model computation.

• Payload Serialization: Time to encode/decode requests/responses (e.g., using Protocol Buffers with gRPC).
• Network Transmission: Round-trip time for data to travel between client and server. Payload size directly impacts this.
• Framework Overhead: Latency from the serving framework's request handling, routing, and response streaming setup.

Latency from the operating system, hardware drivers, and orchestration layer that facilitates the inference work.

• Context Switching: CPU overhead from managing GPU work queues and inter-process communication.
• Kernel Launch: The GPU kernel launch overhead for initiating the prefill and decode operations.
• Orchestration Lag: In cloud environments, delay from the container orchestrator (e.g., Kubernetes) assigning the pod to a node. Related to autoscaling lag for new instances.

TTFT is dominated by the prefill phase, where the entire input prompt is processed in a single, large forward pass to generate the initial Key-Value (KV) cache. Optimizations include:

• Operator Fusion: Compiling the model to fuse sequential operations (e.g., linear, bias, activation) into single GPU kernels, reducing launch overhead.
• Flash Attention: Using optimized attention algorithms that reduce memory reads/writes and improve hardware utilization on long contexts.
• Graph Optimization: Leveraging inference compilers like TensorRT or ONNX Runtime to create a static, optimized execution graph, eliminating runtime decision overhead.

The KV cache for the prompt can consume gigabytes of memory, causing allocation delays or even out-of-memory errors that spike TTFT. Key strategies are:

• PagedAttention: As implemented in vLLM, this treats the KV cache as virtual memory, storing non-contiguous 'pages' in physical memory. This eliminates fragmentation waste from variable-length sequences, allowing higher batch sizes and more predictable memory allocation.
• Quantized Cache: Storing the KV cache in FP8 or INT8 precision (if the model supports it) to halve or quarter its memory footprint, speeding up memory-bound operations.
• Continuous Batching: Dynamically batching incoming requests, ensuring the GPU is fully utilized for the prefill phase rather than processing single requests sequentially.

A cold start occurs when a model is not loaded in GPU memory, adding seconds or minutes to TTFT for the first request. Mitigation involves:

• Model Warmup: Proactively loading high-priority models into memory during system initialization or during periods of low load.
• Pre-allocated Pools: Maintaining a pool of pre-loaded models for anticipated traffic, often managed by orchestration systems like Kubernetes with specialized device plugins.
• Optimized Serialization: Using formats like Safetensors for faster model loading and leveraging NVIDIA's TensorRT-LLM which can persist pre-optimized engine files on disk for rapid deployment.

While primarily for improving Time Per Output Token (TPOT), speculative decoding can reduce effective TTFT in some scenarios by reducing total computation time for short outputs.

• A small, fast draft model (e.g., a distilled version) runs autoregressively to propose a short sequence of candidate tokens.
• The large target model then verifies this sequence in a single, parallel forward pass.
• If the draft is highly accurate, the system produces several tokens in the time it would normally take to produce one, making the first real token from the target model arrive sooner.

Systematic profiling is essential to pinpoint the root cause of high TTFT. Key areas to instrument:

• GPU Kernel Timeline: Use PyTorch Profiler or NVIDIA Nsight Systems to see if prefill is dominated by memory-bound ops (e.g., layer norm) or compute-bound ops (e.g., matrix multiplies).
• CPU/GPU Trace: Identify if time is spent on data serialization, Python-to-kernel launch overhead, or CPU-side pre-processing.
• Memory Allocation: Profile CUDA malloc events to see if KV cache allocation is a significant contributor to latency.
• Comparative Analysis: Profile TTFT across different input lengths to isolate context-processing overhead.

The serving infrastructure itself introduces overhead. Optimizations include:

• gRPC vs. REST: gRPC with HTTP/2 typically offers lower connection overhead and more efficient serialization via Protocol Buffers than JSON-based REST APIs.
• Minimal Payloads: Structuring requests to avoid unnecessary metadata and using efficient tokenizers to keep input payloads small.
• Synchronous vs. Asynchronous: For true streaming responses where TTFT is critical, synchronous request-response patterns are often simpler and provide more predictable initial delivery, whereas asynchronous patterns may introduce queueing complexity.
• Dedicated Hardware: Using inference-optimized GPUs (e.g., NVIDIA L4, H100) with high memory bandwidth and specialized cores (like Tensor Cores for FP8) specifically accelerates the large matrix operations of the prefill phase.

End-to-end latency is the total elapsed time from when a client initiates a request until the complete, final response is received and usable. This is the ultimate user-facing metric.

• Encompasses all components: Network transmission, server-side queuing, prefilling latency, TTFT, Time Per Output Token (TPOT) for the full sequence, and any result post-processing.
• Key distinction: While TTFT measures perceived responsiveness, end-to-end latency measures total task completion time. For long generations, TTFT may be low but end-to-end latency high.

Time Per Output Token (TPOT), also known as inter-token latency, is the average time required to generate each subsequent token after the first one in an autoregressive sequence.

• Directly impacts streaming speed: After TTFT, TPOT determines how quickly the rest of the completion streams to the user.
• Governed by decoding: This phase is memory-bandwidth bound, reading the Key-Value (KV) cache and performing the sampling operation for each new token.
• Throughput relationship: Systems optimized for high Queries Per Second (QPS) often trade off higher TPOT for better overall throughput via techniques like larger batch sizes.

Prefilling latency is the compute time required to process the static input prompt and context through the model's forward pass before any token generation begins. This phase directly precedes and feeds into TTFT.

• Primary contributor to TTFT: For long prompts, prefilling can be the dominant cost of TTFT.
• Creates the KV cache: This phase computes and stores the Key-Value (KV) cache for the prompt tokens, which is then reused during the efficient decoding phase.
• Optimization target: Techniques like FlashAttention and effective continuous batching of prompts are crucial to minimize prefilling latency.

Tail latency refers to the high-percentile response times (e.g., the 95th or 99th percentile) that represent the slowest requests in a distribution. Managing tail TTFT is critical for consistent user experience.

• Causes include: Cold starts, garbage collection pauses, network variability, GPU kernel launch overhead, and resource contention from concurrent requests.
• SLO definition: Service Level Objectives (SLOs) for latency are often defined on P99 or P95 TTFT to guarantee worst-case performance.
• Measurement importance: Average TTFT can mask poor tail performance, which is more perceptually damaging to users.

Continuous batching (or in-flight batching) is a dynamic scheduling technique that adds new inference requests to a running batch as previous requests finish generation, rather than waiting for a whole batch to complete.

• Directly optimizes TTFT & throughput: Maximizes GPU utilization by eliminating idle time, allowing new requests to begin prefilling immediately.
• Reduces queuing delay: A primary technique to lower request queuing delay, a major component of end-to-end latency.
• Core to modern servers: Implemented in engines like vLLM and TensorRT-LLM to handle variable-length sequences efficiently.

A Service Level Objective (SLO) for latency is a formal, measurable target for system reliability and performance. For generative AI, this is often defined as a percentile bound on TTFT or end-to-end latency.

• Example SLO: "P99 TTFT < 500ms for prompts under 1k tokens."
• Basis for engineering decisions: SLOs create an error budget, guiding trade-offs between model quality, cost, and latency.
• Requires robust measurement: Enforced via profiling, canary analysis, and real-time monitoring against a performance baseline.