Time-to-First-Token (TTFT)

The total parameter count directly impacts the time required to load weights into accelerator memory and perform the initial matrix multiplications.

• Parameter Count: Larger models (e.g., 70B vs. 7B) require more FLOPs per token.
• Architecture Type: Mixture-of-Experts (MoE) models activate only a subset of parameters, often yielding lower TTFT than dense models of equivalent total size.
• Quantization Level: Lower precision formats (INT4, INT8) reduce memory bandwidth pressure, accelerating the initial computation.

Before generating the first new token, the model must process the entire input prompt. This prefill phase is compute-bound.

• Prompt Length: TTFT scales quadratically with sequence length due to the self-attention mechanism.
• KV-Cache Initialization: The model computes and stores key-value pairs for every input token, consuming memory bandwidth.
• System Prompts: Long, static system prompts are processed once per session but add to the initial prefill latency.

The physical compute layer dictates the upper bound of inference speed.

• Accelerator Type: NVIDIA H100 GPUs offer significantly higher memory bandwidth (3.35 TB/s) than A100s, reducing TTFT.
• Interconnect: NVLink and InfiniBand bandwidth between GPUs in tensor-parallel deployments prevents communication bottlenecks.
• Batch Saturation: High utilization from concurrent requests can cause queueing delays, increasing TTFT if the scheduler is not optimized for prefill-priority.

The inference server's orchestration logic determines how compute resources are allocated to incoming requests.

• Continuous Batching: Dynamically adding or removing sequences from a batch prevents head-of-line blocking.
• Prefill-Priority Scheduling: Prioritizing the prefill phase for new requests over decoding steps for ongoing ones minimizes TTFT.
• Disaggregated Serving: Physically separating prefill and decode servers allows independent scaling and resource allocation for each phase.

The physical distance between the user and the inference endpoint adds a fixed, irreducible delay.

• Round-Trip Time (RTT): A user in Singapore querying a server in Virginia incurs ~250ms of unavoidable latency.
• Edge Deployment: Placing inference servers at the network edge (CDN-like) minimizes RTT.
• Connection Setup: TLS handshake and HTTP/2 or WebSocket connection establishment add one-time overhead to the first request.

The text-to-token conversion and input formatting pipeline adds a small but measurable delay.

• Tokenizer Speed: Byte-Pair Encoding (BPE) tokenizers are generally fast, but complex Unicode normalization can add latency.
• Chat Template Application: Formatting raw text into the model's specific chat template (e.g., applying <|im_start|> tokens) requires CPU processing.
• Input Validation: Guardrails, content filtering, and prompt injection checks performed before inference add to the pre-generation timeline.

A serving technique where generated tokens are transmitted to the client sequentially as they are produced, rather than waiting for the full response. This directly minimizes the perceived latency of the model by allowing users to see the beginning of the response immediately after the Time-to-First-Token (TTFT) interval elapses. Streaming is essential for maintaining conversational flow in chat interfaces.

The specific mechanism of delivering partial LLM outputs token-by-token over a persistent HTTP/2 or WebSocket connection. Unlike batch inference, token streaming enables real-time text display in chat interfaces. The interval between successive tokens is measured as Time-Per-Output-Token (TPOT) , which, combined with TTFT, defines the total user-facing latency profile.

An inference optimization technique that dynamically adds new requests to a running batch without waiting for all prior sequences to complete. This dramatically improves GPU utilization and reduces queuing delays. By processing multiple requests concurrently, continuous batching directly lowers the Time-to-First-Token for queued requests, as they no longer wait for a full batch to finish before generation begins.

The practice of structuring prompts and conversation history to fit within a model's token limits without sacrificing critical information. A bloated system prompt or excessive conversation history increases prefill latency—the time required to process the input before the first token is generated. Optimizing context directly reduces TTFT by minimizing the computational cost of the initial forward pass.

A generation technique that forces the model's output to strictly adhere to a predefined schema, grammar, or regular expression. While this ensures structural validity for API calls or JSON outputs, it can increase inter-token latency as the system validates each candidate token against the constraint. The impact on TTFT is typically minimal, but the overall generation speed may be affected by the complexity of the grammar.

The initial stage of LLM inference where the entire input prompt is processed in parallel to compute the Key-Value (KV) cache. This phase is compute-bound and its duration is directly proportional to the input length. The Time-to-First-Token is the sum of the prefill phase duration plus the time to sample and emit the first output token. Reducing input size is the primary lever for lowering TTFT.