Continuous Batching

Unlike static batching where the batch size is fixed for the entire generation, continuous batching allows the batch composition to change mid-execution. As some sequences within the batch finish generation (by producing an end-of-sequence token), their slots are immediately filled with new, pending requests. This eliminates idle GPU cycles and ensures the compute hardware is constantly saturated with work, dramatically improving aggregate throughput.

The primary goal is to keep the Tensor Cores and memory bandwidth of expensive GPUs (e.g., NVIDIA H100, A100) as busy as possible. Static batching suffers from low utilization during the decode phase, as shorter sequences finish early and their GPU resources sit idle. Continuous batching maintains near-peak FLOPs utilization by continuously feeding new computational work into the pipeline, often achieving 2-5x higher throughput for mixed-length requests compared to naive batching.

Execution is managed at the granularity of a single decoding iteration (producing one token per sequence). The scheduler, after each iteration:

• Identifies finished sequences.
• Evicts them from the batch.
• Selects new requests from a queue.
• Dynamically updates the Key-Value (KV) Cache in memory to accommodate the new sequences. This fine-grained control is what enables the 'continuous' aspect, making it highly responsive to fluctuating request loads.

By allowing new requests to join a batch immediately, continuous batching significantly lowers queue time for users. In a static system, a request must wait for the next batch to be formed, which could be hundreds of milliseconds away. Continuous batching can start processing requests within milliseconds, improving Time to First Token (TTFT) for most users and creating a more responsive experience, especially under variable load.

This technique requires sophisticated management of the KV Cache, which stores attention key/value vectors for all previous tokens in each sequence. The system must:

• Dynamically allocate and deallocate memory for sequences as they enter and leave the batch.
• Implement paged attention (as seen in vLLM) to handle non-contiguous memory blocks efficiently.
• Avoid memory fragmentation to sustain high performance. This memory orchestration is a core engineering challenge and differentiator between serving systems like vLLM, TGI, and TensorRT-LLM.

Static (Traditional) Batching:

• Fixed set of requests processed together.
• Batch waits for the slowest sequence to finish.
• Low GPU utilization during decode.
• Predictable but poor latency/throughput trade-off.

Continuous (Iteration) Batching:

• Batch composition changes every decoding step.
• No waiting for the slowest sequence; new work fills gaps.
• High, sustained GPU utilization.
• Optimal latency/throughput trade-off for online serving. This makes it the de facto standard for production LLM serving APIs.

Static batching is the predecessor to continuous batching, where inference requests are grouped into a fixed-size batch and processed simultaneously. The entire batch must complete generation before any results are returned and a new batch can begin.

• Key Limitation: Causes high tail latency, as fast-generating requests are held up waiting for slower ones in the same batch.
• GPU Utilization: Often leads to low GPU utilization during the decode phase, as the number of active sequences decreases over time.
• Contrast: Continuous batching dynamically adds new requests to fill these idle compute slots, solving the core inefficiency of static batching.

Iteration-level scheduling is the underlying scheduling paradigm that enables continuous batching. Instead of scheduling entire requests, the system schedules individual decoding iterations for each sequence.

• Mechanism: On each cycle, the scheduler identifies all sequences that are ready to generate their next token (i.e., have received their previous token). These sequences are packed into a new batch for that single forward pass.
• Flexibility: Allows new requests to join the batch and finished requests to exit on every iteration, creating a fluid, continuously processing batch.
• Implementation: This fine-grained scheduling is the core innovation in systems like NVIDIA's TensorRT-LLM and vLLM's PagedAttention scheduler.

PagedAttention is a memory management algorithm for the KV Cache that is foundational for efficient continuous batching. It applies concepts from operating system virtual memory to LLM serving.

• Problem: Traditional KV cache allocation is monolithic and inflexible, causing memory fragmentation and limiting batch size when sequences finish at different times.
• Solution: PagedAttention divides the KV cache into fixed-size blocks. Sequences can store their keys and values in non-contiguous blocks, just as processes use pages in physical memory.
• Impact: Enables highly efficient sharing of GPU memory between concurrent sequences, allowing continuous batching systems to maintain very large batch sizes with diverse sequence lengths. It is the engine behind vLLM's high throughput.

Time to First Token is a critical user-facing latency metric that measures the delay from submitting a request to receiving the first token of the output. Continuous batching directly impacts TTFT.

• Prefill Phase: TTFT is dominated by the prefill stage, where the entire input prompt is processed in one forward pass. In continuous batching, a new request may wait briefly for the next scheduling iteration before its prefill can begin.
• Trade-off: Aggressive continuous batching prioritizes high overall throughput (Tokens per Second) by packing batches fully, which can slightly increase queue time and thus TTFT for individual requests.
• Optimization: Advanced schedulers may prioritize requests in interactive scenarios to minimize TTFT, even at a slight cost to overall throughput.

Tokens per Second is the primary throughput metric for LLM inference, measuring the total output tokens generated by the system per second. Continuous batching is the most effective technique for maximizing TPS.

• Goal: Maximize GPU utilization during the long decode phase by ensuring the GPU is never idle.
• Achievement: By dynamically filling the batch with new sequences as others finish, continuous batching can achieve near-100% GPU utilization during decoding, leading to 5-10x higher TPS compared to static batching for workloads with variable request rates and sequence lengths.
• Measurement: TPS is typically measured under a specific load pattern and is the key business metric for cost-per-token calculations.

Orchestration frameworks provide the production infrastructure to deploy and scale models using techniques like continuous batching. They abstract away the low-level scheduling complexity.

• Ray Serve: A scalable model-serving library built on Ray. It supports continuous batching via its max_batch_size and batch_wait_timeout_s parameters, dynamically batching requests across replicas.
• Hugging Face Text Generation Inference (TGI): A dedicated toolkit for deploying LLMs. It implements continuous batching with custom CUDA kernels and token streaming, supporting popular open-source models.
• Function: These frameworks handle request queuing, model replication, health checks, and integration of the continuous batching scheduler, allowing engineers to focus on application logic rather than low-level optimization.