Prefilling Latency

Prefilling latency is incurred once per request to process the static input prompt and context. Unlike the iterative decoding latency for each output token, this phase is non-recurrent. The generated Key-Value (KV) Cache is stored and reused for all subsequent autoregressive generation steps, making this initial cost amortized over the length of the output. This characteristic makes optimizing the prefill phase particularly impactful for short, conversational outputs.

The computational cost of the prefill forward pass scales quadratically with the input sequence length due to the self-attention mechanism. For a prompt of length N, the attention operation has O(N²) complexity. This makes long-context prompts (e.g., 128k tokens) extremely expensive to prefill, often dominating total inference time. Techniques like FlashAttention are critical for managing this complexity by optimizing memory access patterns on hardware like GPUs.

Time to First Token (TTFT) is the user-facing metric most directly determined by prefill latency. In a streaming response setup, the client perceives TTFT as the wait time before the first word appears. Since token generation cannot begin until the KV cache is populated, prefilling latency is the lower bound for TTFT. Reducing prefill time is therefore essential for improving perceived responsiveness in interactive applications like chatbots.

The prefill phase is often memory-bandwidth bound, especially for large models. The process involves loading the entire set of model weights from GPU High-Bandwidth Memory (HBM) to compute the initial forward pass. The massive size of model parameters (e.g., 70B parameters) means memory bandwidth, not just FLOPs, is a key bottleneck. Optimization strategies focus on:

• Operator fusion to reduce intermediate memory writes.
• Kernel optimization for efficient memory access.
• Model quantization (e.g., FP16, INT8) to reduce the total data moved.

In a production serving system using continuous batching, prefill requests for new prompts are dynamically interleaved with the decoding steps of existing requests. This introduces scheduling complexity. The system must decide when to pause decoding to compute a prefill for a new user, potentially increasing the decoding latency for existing requests. Efficient schedulers aim to batch multiple prefill requests together to maximize GPU utilization while minimizing the stall time for ongoing generations.

It is crucial to distinguish prefilling latency from cold start latency. Prefilling is a per-request computational step. Cold start latency is a per-container or per-pod infrastructure delay that occurs when a model must be loaded from disk into GPU memory to serve the first request after a scale-up or restart. A system can have optimal prefill latency but still suffer from high tail latency due to cold starts if autoscaling is not properly tuned.

Time to First Token (TTFT) is the duration from the start of an inference request to when the first token of the output is generated or delivered to the client. This metric is dominated by prefilling latency but also includes any initial queuing or system overhead. It is the primary determinant of perceived responsiveness in streaming chat applications.

Decoding latency is the time consumed during the autoregressive token generation phase, where each new token is produced conditioned on all previously generated tokens. This is measured after prefilling completes. Time Per Output Token (TPOT) is the average decoding latency per token. Key factors influencing decoding speed include:

• Model size and computational complexity.
• Hardware memory bandwidth for reading the KV cache.
• Batch size and scheduling efficiency.

The Key-Value (KV) Cache is a memory structure that stores the intermediate key and value vectors from the transformer's attention layers for the static prompt context. Prefilling latency is the computational cost of populating this cache. Efficient KV cache management is crucial for performance:

• PagedAttention (vLLM) reduces memory fragmentation for variable-length sequences.
• Cache size grows linearly with batch size and sequence length.
• Cache hits on repeated prompts can bypass prefilling, eliminating its latency.

Continuous batching is an inference optimization technique where new requests are dynamically added to a running batch as previous requests finish generation. This maximizes GPU utilization and throughput. Its interaction with prefilling is critical:

• Incoming requests with new prompts must wait for a prefilling slot on the GPU.
• Efficient schedulers interleave prefilling (compute-heavy) with decoding (memory-bandwidth-heavy) workloads.
• Poor batching can lead to head-of-line blocking, where a long prefilling job delays decoding for other requests.

Cold start latency is the additional delay incurred when the first request arrives at a model instance that is not loaded in memory. This is a superset of prefilling latency and includes:

• Loading model weights from disk or network storage into GPU memory.
• Initializing runtime frameworks (e.g., PyTorch, TensorRT).
• Performing the initial prefilling forward pass. Strategies to mitigate this include pre-warming instances and using model keep-alive policies.

Operator fusion is a compiler optimization that combines multiple sequential neural network operations into a single GPU kernel. This reduces kernel launch overhead and intermediate memory transfers. During the prefilling phase, fusing operations in the attention and feed-forward layers can significantly reduce latency. Frameworks like TensorRT and ONNX Runtime perform this automatically by creating an optimized model execution graph, which is a static, pre-compiled computation plan for the entire forward pass.