Time To First Token (TTFT)

The prefill phase computes the initial key-value pairs for the entire input prompt in parallel. TTFT is directly proportional to the prompt's token length and the model's parameter count. A 10k-token system prompt will inherently have a higher TTFT than a 10-token query due to the massive matrix multiplications required to populate the KV cache before the first token can be decoded.

Cold starts dramatically impact TTFT. The time required to load model weights from disk or network storage into GPU VRAM is a fixed overhead. Techniques like model pre-warming (keeping the model resident in memory) and using fast storage (NVMe SSDs) are essential. The GGUF format and memory-mapped loading in runtimes like llama.cpp minimize this by loading only required tensors on demand.

The inference server's scheduler determines when a new request gets compute time. Continuous batching allows requests to join a running batch immediately, reducing queue wait time. In contrast, static batching forces new requests to wait for the entire current batch to finish. The scheduling algorithm must balance maximizing GPU utilization with minimizing queue delay to keep TTFT low under concurrent load.

In distributed inference, the physical distance between the client and the server adds round-trip time (RTT). Additionally, transferring large prompt payloads over constrained networks adds transmission delay. Deploying inference endpoints on edge architectures or within the same local network via self-hosted LLMs eliminates this variable, providing deterministic, sub-millisecond network overhead for the initial request.

If the existing KV cache is near its memory limit, the server must evict old blocks or wait for memory to free up before processing a new prompt's prefill. This memory management overhead, addressed by algorithms like PagedAttention, directly stalls the start of computation. Insufficient VRAM for the combined KV cache and model weights forces costly offloading to slower CPU RAM, catastrophically increasing TTFT.

Lower numerical precision (e.g., 4-bit quantization) reduces the memory bandwidth required to read model weights during the prefill phase. While this accelerates computation, aggressive quantization can introduce overhead from dequantization kernels. The optimal quantization level (e.g., Q4_K_M in GGUF) balances the speed of reduced memory movement against the computational cost of casting types, directly influencing the time to process the prompt.