Time-to-First-Token (TTFT)

TTFT directly governs the user's perception of system speed. A low TTFT creates a feeling of immediate responsiveness, while a high TTFT signals a sluggish, unresponsive system.

• Sub-200ms: Feels instantaneous. The user perceives no delay.
• 200ms - 1 second: A noticeable but acceptable pause. The user remains engaged.
• > 2 seconds: The user's flow of thought is interrupted, leading to frustration and potential abandonment.

This metric is distinct from total generation time, as users are willing to wait for a complete answer if the system has already begun responding.

It is crucial to decouple TTFT from the total time to generate the full response. A system can have an excellent TTFT but a slow generation rate, or vice-versa.

• TTFT: Measures the initial wait. Dominated by prompt processing, network transit, and input evaluation.
• Total Latency: Measures the time to the last token. Dominated by the full autoregressive decode loop and the number of output tokens.

Optimizing for TTFT often involves different strategies than optimizing for total throughput, such as speculative retrieval or aggressive KV-Cache pre-filling.

In transformer-based models, TTFT is dominated by the prefill phase. During prefill, the entire input prompt is processed in a single, highly parallelizable forward pass to populate the initial state of the KV-Cache.

• Compute-Bound: For very long system prompts or large context windows, the prefill computation can become the primary latency bottleneck.
• Memory-Bound: Loading large model weights from GPU memory also contributes significantly.

Techniques like continuous batching and FlashAttention are critical for minimizing this prefill time to deliver a fast first token.

TTFT is the enabler of token-by-token streaming, the standard user interface pattern for large language models. Without a low TTFT, streaming provides no benefit.

• Progressive Disclosure: Streaming allows the user to begin reading and mentally processing the response while it's still being generated.
• Cancellation: A fast first token allows the user to quickly assess if the response is on the right track and cancel early if not, saving compute resources.

Protocols like Server-Sent Events (SSE) or gRPC Streaming are used to push tokens to the client as soon as they are generated, making TTFT the key metric for the start of this stream.

Accurate measurement requires instrumenting the serving stack from the moment the request is received to the moment the first token is flushed to the client.

• Server-Side Metric: Measured on the inference server, excluding client-side rendering time.
• Percentiles Matter: Track P50, P95, and P99 TTFT. The average can hide tail latency caused by queuing or garbage collection.
• SLO Definition: A robust Service Level Objective (SLO) for TTFT might be: 'P95 TTFT will be < 450ms over a 30-day rolling window.'

Monitoring TTFT alongside cache hit ratio and queue depth helps diagnose the root cause of latency spikes.

Reducing TTFT requires a multi-layered approach across infrastructure and model architecture.

• Speculative Retrieval: Pre-fetch documents predicted to be needed before the query is fully formed.
• Semantic Cache: Serve a cached response immediately if a semantically identical query has been asked before, bypassing inference entirely.
• Prompt Compression: Reduce the size of the input prompt by summarizing or pruning non-essential context.
• Hardware Acceleration: Leverage Neural Processing Units (NPUs) or optimized GPU kernels like FlashAttention-3 to accelerate the prefill computation.
• Load Balancing: Distribute requests intelligently to avoid queuing on overloaded instances, using patterns like backpressure.