Continuous Batching

Unlike static batching, which waits for a fixed batch size, continuous batching dynamically groups incoming requests into a shared execution context. This allows the system to start processing new requests immediately, even while previous batches are still generating tokens. This is the core mechanism that eliminates idle GPU time and improves throughput SLIs.

The scheduler operates at the granularity of a single generation iteration (one forward pass). After each iteration, completed sequences are removed from the batch, and newly arrived or paused requests are added. This fine-grained control is what enables the continuous flow of work, maximizing GPU utilization and directly improving Time Per Output Token (TPOT) metrics.

Continuous batching is enabled by efficient memory management systems like Paged Attention (used in vLLM). This technique manages the Key-Value (KV) cache in non-contiguous, paged blocks, similar to virtual memory in operating systems. This allows for:

• Flexible sharing of GPU memory across sequences of different lengths.
• Elimination of internal fragmentation from padding.
• Efficient swapping of paused sequences, which is critical for handling long contexts and variable request lifecycles.

By eliminating the queue time associated with waiting for a full static batch, continuous batching dramatically reduces Time To First Token (TTFT) for individual requests. This improves user-perceived responsiveness and helps meet stringent percentile latency SLOs (e.g., p95, p99). The technique is particularly effective for interactive applications like chatbots, where low initial latency is critical.

Continuous batching natively handles sequences with different prompt lengths, generation lengths, and completion states within the same batch. This is a fundamental advantage over static batching, which requires padding all sequences to the length of the longest one in the batch, wasting significant compute and memory. This efficiency directly translates to lower cost per query and higher overall system throughput.

The traditional inference batching method where requests are grouped into fixed-size batches, and the entire batch must complete processing before the next batch begins. This leads to significant GPU idle time as faster requests wait for the slowest request in the batch to finish, creating a straggler effect. It is inefficient for variable-length sequences common in language model inference.

A critical latency SLI for interactive AI services, measuring the duration from request submission to the generation of the first output token. Continuous batching directly impacts TTFT by minimizing queue time; requests can begin execution immediately upon arrival into a running batch rather than waiting for a new static batch to form. Optimizing for TTFT often involves trade-offs with total throughput.

A throughput SLI measuring the average latency to generate each subsequent token after the first. Continuous batching optimizes TPOT by maintaining high GPU utilization throughout the streaming phase. As some requests finish and new ones start, the computational load remains balanced, leading to a higher aggregate token generation rate across all concurrent requests, which is essential for cost-efficiency SLOs.

The subsystem responsible for storing intermediate attention computations during autoregressive generation. Efficient KV Cache management is foundational for continuous batching. Techniques include:

• PagedAttention (vLLM): Eliminates internal fragmentation.
• Dynamic KV Cache allocation: Allocating and freeing memory per request as needed.
• Cache eviction policies: For handling very long contexts. Poor management leads to out-of-memory errors or severely limited batch sizes, breaking throughput SLOs.