Time to First Token (TTFT)

Before the model begins inference, the system must ingest and prepare the request. This includes network transmission, API gateway routing, request validation, and prompt tokenization—the process of converting the input text into the numerical tokens the model understands. Latency here is influenced by network proximity, gateway efficiency, and the length/complexity of the input prompt.

For very large models, the weights may be partially offloaded from GPU memory. TTFT includes the time to load necessary model parameters into active memory (GPU RAM). It also includes initializing the KV (Key-Value) cache—a memory structure that stores computed attention states for the input sequence, which is essential for efficient autoregressive generation. Cold starts, where no model is loaded, result in significantly higher TTFT.

This is the most computationally intensive part of TTFT. The entire input prompt is processed in a single, parallel forward pass through the model. The system computes:

• Attention scores across the full context window.
• The hidden state for the final token position, which becomes the starting point for generation. This phase's duration scales linearly with the number of input tokens and is bound by GPU compute speed and memory bandwidth.

After the prefill phase, the model uses the final hidden state to generate a probability distribution (logits) over its vocabulary. The system then applies a sampling strategy (e.g., greedy, top-p, temperature) to select the first output token. This involves:

• Logit computation for the vocabulary.
• Sampling algorithm execution.
• Token decoding (converting the token ID back to a string or byte sequence).

Once the first token is selected, it must be sent to the client. This involves:

• Response serialization into the API protocol (e.g., JSON over HTTP/SSE).
• Network transmission time (latency).
• Client-side buffering before the application can render the token. For Server-Sent Events (SSE) or similar streaming protocols, this is typically minimal but non-zero.

In a shared serving environment, requests may not be processed immediately. TTFT includes time spent in a scheduler queue waiting for an available GPU instance or batch slot. Advanced systems use continuous batching, which can group multiple requests for efficient prefill, but a request may still wait for the next batching cycle. This is a major source of TTFT variance under load.

The total elapsed time from a user's initial request to the receipt of the agent's final, actionable output. This encompasses all sub-processes:

• TTFT (initial processing delay)
• Time Per Output Token (generation speed)
• Network transmission
• External tool or API call execution
• Post-processing and delivery. It is the primary metric for user-perceived responsiveness.

A throughput metric measuring the rate of token generation after the first token is received. It directly impacts the Time Per Output Token and the overall flow of a streaming response.

• High TPS reduces total completion time for long outputs.
• It is often in tension with TTFT; optimization for one can negatively affect the other.
• Measured at the inference engine level, distinct from end-user perceived speed.

Measures the worst-case response times experienced by a small fraction of requests, critical for understanding service reliability.

• P95 Latency: 95% of requests are faster than this value.
• P99 Latency: 99% of requests are faster than this value; captures extreme outliers.
• High TTFT at the P99 can indicate resource contention, cold starts, or unpredictable bottlenecks, severely degrading user experience for a subset of interactions.

The number of simultaneous requests or agent sessions a system is processing. It is a primary driver of latency metrics.

• Under low concurrency, TTFT is often optimal.
• As concurrency increases, systems may experience:
  • Queueing delays before processing begins (increasing TTFT).
  • Resource sharing (GPU/CPU) slowing down token generation (reducing TPS).
• Defines the system's operational capacity before saturation.

A target for reliability or performance that is derived from Service Level Indicators (SLIs) like TTFT or end-to-end latency.

• Example: "99% of agent requests shall have a TTFT of < 500ms over a 30-day rolling window."
• TTFT SLOs are common for interactive agent applications to guarantee perceived responsiveness.
• Breaching an SLO consumes the Error Budget, which guides release and prioritization decisions.

The component or resource within a system that limits overall throughput or increases latency. For TTFT, common bottlenecks include:

• Cold Model Loading: Loading weights into GPU memory for the first request.
• Prompt Processing: The computational cost of encoding a long context window.
• Orchestration Overhead: Delays introduced by agent frameworks in routing and planning before inference begins.
• Identifying and mitigating the TTFT bottleneck is a key optimization task.