Continuous Batching

Unlike static batching, which waits for a fixed batch to fill, continuous batching maintains a running batch of active requests. New requests are added to this pool as soon as they arrive, and finished requests are removed immediately upon generating their end-of-sequence token. This creates a fluid batch size that adapts to the real-time request load, preventing idle compute cycles and dramatically improving GPU utilization.

Scheduling decisions are made at the granularity of a single generation step (iteration), not per request. The scheduler examines all active requests in the pool before each forward pass of the model. It groups only the tokens that need to be processed in the next step. This fine-grained control is the core mechanism that allows efficient interleaving of requests at different stages of completion, minimizing padded tokens and wasted FLOPs.

A key engineering challenge is managing the Key-Value (KV) cache for requests of variable sequence lengths within a dynamic batch. Advanced systems like vLLM use PagedAttention, which manages the KV cache in non-contiguous, paged memory blocks (similar to virtual memory in operating systems). This allows:

• Efficient memory sharing for identical prompts.
• Elimination of internal fragmentation from padding.
• Dynamic allocation and deallocation as requests start and finish.

Continuous batching provides a tunable knob between latency and throughput. Parameters like the maximum batch size and scheduling policy (e.g., First-Come-First-Served vs. prioritizing short requests) can be adjusted. This allows serving systems to be optimized for different scenarios:

• Low-latency interactive chat: Smaller, more fluid batches.
• High-throughput batch processing: Larger, more stable batches. The system inherently provides better latency for early-finishing requests compared to static batching.

Continuous batching is not a standalone technique; it combines synergistically with other inference optimizations:

• Quantization: Lower precision weights (e.g., INT4) reduce memory bandwidth, allowing larger dynamic batches.
• FlashAttention: Optimized attention computation works within the dynamic batch structure to handle variable sequence lengths efficiently.
• Speculative Decoding: The draft and verification stages can be scheduled across the dynamic request pool. Frameworks like TensorRT-LLM and vLLM integrate these techniques into a cohesive optimization pipeline.

Continuous batching is implemented in several high-performance open-source inference servers and frameworks, each with specific advantages:

• vLLM: Uses PagedAttention for best-in-class memory efficiency and high throughput.
• TensorRT-LLM: NVIDIA's solution offering deep kernel fusion and optimization for NVIDIA GPUs alongside continuous batching.
• TGI (Text Generation Inference): Hugging Face's server, supporting popular models with continuous batching.
• SGLang: A newer framework that extends continuous batching for complex LLM programming patterns like chained reasoning and agent loops.

These are newer, high-performance frameworks that implement advanced continuous batching strategies.

• SGLang (from LMSYS) is a co-design of frontend language and backend runtime. It uses RadixAttention, a technique that automatically caches and reuses the KV cache of common prefixes across requests (e.g., shared system prompts), which is a form of semantic-level batching optimization.
• LMDeploy (from InternLM) is a toolkit for compressing, deploying, and serving LLMs. Its serving engine, Turbomind, features continuous batching with a persistent batch scheduler and efficient W4A16 kernel support, optimized for NVIDIA Tensor Cores.

Continuous batching is an implementation of a broader systems concept called iteration-level scheduling or rolling batching. The core mechanism is:

• The batch is not static; it is a dynamic set of active sequences.
• The scheduler operates at the granularity of a single generation step (iteration).
• After each step, completed sequences are evicted, and new, eligible requests are inserted.
• This maximizes GPU utilization by ensuring the processing units are always fed with work, minimizing idle time between fixed batch executions. This is contrasted with static batching (fixed for the entire generation) and basic dynamic batching (batches formed only at request arrival).

Static batching is an inference strategy where a fixed number of input samples are grouped into a batch before processing. All requests in the batch must start and finish generation together.

• Key Limitation: Poor GPU utilization when requests have different sequence lengths, as the entire batch waits for the slowest request to finish.
• Contrast with Continuous Batching: Serves as the baseline inefficient method that continuous batching was designed to improve upon by eliminating idle time.

Dynamic batching is an inference optimization where a serving system groups incoming requests into batches on-the-fly, but typically only at request boundaries.

• Operation: The system waits a short time (a scheduling window) to collect multiple requests, then processes them as a single batch.
• Limitation: While better than static batching, it still suffers from inefficiency for autoregressive text generation, as generated tokens cannot be added to a running batch. Continuous batching is a specialized form of dynamic batching that operates at the token level.

PagedAttention is an algorithm for managing the Key-Value (KV) cache in large language model inference, introduced by the vLLM system. It is a critical enabler for efficient continuous batching.

• Mechanism: Manages the KV cache in non-contiguous, fixed-size memory blocks (pages), analogous to virtual memory in operating systems.
• Benefit for Continuous Batching: Allows for flexible memory sharing between sequences and eliminates internal fragmentation. This enables new requests to be added to a batch without reallocating the entire cache, which is essential for the rolling nature of continuous batching.

Inference latency is the time delay, measured in milliseconds, between submitting an input to a model and receiving its output. Continuous batching has a complex relationship with this critical metric.

• Throughput vs. Latency Trade-off: Continuous batching is primarily a throughput-maximizing technique. It can increase Time To First Token (TTFT) for individual requests as they may wait briefly in a scheduler queue.
• Optimization Goal: The scheduler's objective is to maximize overall GPU utilization (throughput) while keeping added latency within service-level agreement (SLA) bounds.

Iteration-level scheduling is a synonym for continuous batching that emphasizes its operational granularity.

• Granularity: Scheduling decisions (adding new requests, removing finished ones) are made at each decoding iteration (step), not just at the start or end of a request.
• System View: This term highlights the continuous batching scheduler as a core component of the inference server's runtime, which must manage computation, memory, and context swaps at every step of the generation process for all concurrent sequences.