Continuous Batching

Unlike static batching, which waits for all sequences in a batch to finish before accepting new work, continuous batching evaluates the batch state at every forward pass iteration. As soon as a sequence generates an end-of-sequence (EOS) token, it is evicted from the batch, and a new sequence from the waiting queue is immediately inserted. This prevents GPU cores from idling while waiting for straggler sequences with long outputs.

Efficient continuous batching requires intelligent memory management of the KV Cache. Since sequences enter and exit the batch dynamically, the memory allocated for their key-value tensors must be rapidly reclaimed and reassigned. Advanced implementations use PagedAttention to manage this cache in non-contiguous blocks, eliminating internal fragmentation and allowing the system to pack more concurrent sequences into GPU VRAM.

To prevent a single long-running generation from monopolizing the batch, continuous batching systems implement preemption policies. If a new high-priority request arrives or a sequence exceeds a token budget, the system can temporarily swap its KV cache blocks to host memory (CPU RAM) and re-admit it later. This ensures low Time To First Token (TTFT) for all users under heavy load.

Modern continuous batching engines often split the prefill phase (processing the prompt) from the decode phase (generating tokens). Prompts are compute-bound and processed at maximum throughput, while token generation is memory-bound. By scheduling these phases independently, the system can interleave high-priority prefill operations with ongoing decodes, maximizing hardware utilization without starving interactive users.

The primary goal of continuous batching is to saturate GPU Tensor Cores and memory bandwidth. By ensuring the batch size remains at the maximum possible limit without causing out-of-memory errors, the system achieves near-peak Tokens Per Second (TPS). This directly translates to lower cost-per-token for providers and higher throughput for enterprise deployments.

A memory buffer that stores computed key and value tensors from previous tokens during autoregressive generation. Without a KV cache, each new token would require recomputing attention over the entire sequence, making generation quadratically expensive. The KV cache is the primary memory bottleneck in LLM serving, and its efficient management is the central challenge that continuous batching and paged attention aim to solve.

An inference acceleration method that uses a smaller draft model to generate candidate tokens quickly, which are then verified in parallel by a larger target model. While continuous batching optimizes across requests, speculative decoding optimizes within a single sequence. The two techniques are complementary and can be combined for multiplicative throughput gains in production serving systems.

An optimization that stores and reuses the computed KV cache for a shared prompt prefix across multiple generation requests. In scenarios like chatbots with long system prompts or few-shot examples, prefix caching prevents redundant computation. When combined with continuous batching, it allows the scheduler to route requests with identical prefixes to the same batch, dramatically reducing Time To First Token.

A critical latency metric measuring the delay between sending a prompt to an inference server and receiving the first generated token. Continuous batching directly impacts TTFT by allowing new requests to join a running batch immediately rather than waiting for the current batch to complete. This is the primary user-experience metric that continuous batching optimizes, as it determines how quickly a user sees the model begin responding.