Continuous Batching

The core mechanism of continuous batching is its ability to interleave the processing of multiple requests within the same batch. Unlike static batching, which processes a fixed set of prompts from start to finish, continuous batching allows the inference engine to:

• Add new requests to an active batch as soon as GPU resources are available.
• Remove completed sequences (where an end-of-sequence token is generated) without stopping the batch.
• This creates a pipeline of requests at different stages of completion, maximizing hardware utilization by ensuring the GPU is never idle waiting for a single slow request to finish.

Scheduling decisions are made at the granularity of a single decoding iteration (generating one token per sequence), not per request. The scheduler maintains a global batch of active sequences. Each iteration:

1. The model's forward pass computes the next token for every sequence in the active batch.
2. Sequences that have finished generation are identified and evicted.
3. Pending requests from an input queue are selected and added to the now-available slots. This fine-grained control is what enables high throughput, as it minimizes the time any computational unit (e.g., a GPU tensor core) is left underutilized.

Continuous batching is intrinsically linked to efficient Key-Value (KV) Cache management. Each sequence in the batch has its own growing KV cache. Advanced systems like vLLM with PagedAttention are essential companions because they:

• Allow the KV cache for different sequences to be stored in non-contiguous memory blocks.
• Enable shared memory for identical prompt prefixes across requests (e.g., a system prompt).
• Dynamically allocate and deallocate cache pages as sequences are added and removed, preventing memory fragmentation. Without such efficient cache management, the memory overhead of many concurrent sequences would negate the throughput benefits of continuous batching.

The primary operational benefit is a dramatic increase in GPU utilization and tokens/second throughput, especially under variable or interactive workloads. Key metrics include:

• Time to First Token (TTFT): Often improved for later-arriving requests, as they don't wait for a new batch to form.
• Inter-Token Latency: Remains stable per request but is amortized across many concurrent requests.
• Throughput Gain: Can be 4x to 10x+ higher than static batching for workloads with variable sequence lengths and arrival times, as it keeps the computational saturation of the GPU consistently high.

Continuous batching excels in real-world, production inference scenarios where request patterns are unpredictable. It is inherently adaptable to:

• Interactive Chat: Users sending messages at different times, creating an irregular stream of short requests.
• Mixed Request Types: A blend of long document summarization tasks and short classification queries.
• Variable Sequence Lengths: Requests that generate outputs of vastly different lengths without forcing the batch to wait for the longest one. This adaptability makes it the default choice for model serving engines like vLLM, TensorRT-LLM, and TGI (Text Generation Inference) that power modern LLM APIs.

Understanding continuous batching is clarified by contrasting it with static batching:

Static batching is simpler but leads to head-of-line blocking, where fast requests are delayed by slower ones in the same batch.

PagedAttention is the memory management algorithm that makes continuous batching efficient. It organizes the KV cache into non-contiguous, fixed-size blocks (pages), similar to virtual memory in an operating system. This allows for:

• Zero internal fragmentation: Memory waste from variable-length sequences is eliminated.
• Efficient memory sharing: Pages can be shared between sequences in a batch when there is shared prefix (e.g., a common system prompt).
• Dynamic allocation: New pages are allocated on-demand as sequences grow, and freed immediately when a request finishes. This is the foundational technology behind high-throughput inference engines like vLLM.

The Key-Value (KV) Cache is the critical memory structure that continuous batching optimizes. During the autoregressive decode phase of a transformer model, the computed key and value tensors for previous tokens are stored to avoid recomputation. Its management directly dictates performance:

• Memory Bound: Inference is often limited by the speed of reading this cache, not by compute.
• Capacity Limits Context: The size of the allocated KV cache defines the maximum context window.
• Optimization Target: Techniques like KV cache quantization, compression, and offloading aim to reduce its footprint, enabling larger batches or longer contexts.

Transformer inference has two distinct computational phases that continuous batching must handle:

• Prefill Phase: The model processes the entire input prompt in parallel. This is compute-intensive (FLOPs-bound) and generates the initial KV cache. In a continuous batch, new requests trigger this phase.
• Decode Phase: The model generates tokens one-by-one, using the KV cache for efficient attention over the cached context. This phase is memory-bandwidth bound, as it repeatedly reads the cache. Continuous batching maximizes GPU utilization by ensuring the GPU is always busy with either prefill (for new requests) or decode (for ongoing generations) work.

Continuous batching is a scheduling algorithm within a larger model serving system. These architectures handle the full lifecycle:

• Request Queuing: Incoming prompts are held in a queue until the scheduler (using continuous batching) groups them.
• Dynamic Resource Allocation: GPU memory and compute are allocated per-request based on its context length.
• Orchestration: Systems like vLLM, TensorRT-LLM, and proprietary solutions manage the interplay between continuous batching, KV cache management, and hardware execution. The goal is to provide high throughput and low latency at the lowest possible inference cost.

Continuous batching is particularly effective because LLM inference is often memory-bound. This means system performance is limited by the speed of memory accesses (reading/writing the KV cache and model weights), not by the GPU's raw computational power (FLOPs).

• Bottleneck: The time to read the KV cache for attention dominates the decode step.
• Optimization Goal: Techniques aim to reduce the volume of data moved (via quantization) or improve access patterns (via PagedAttention).
• Implication: Simply using a faster GPU may not improve throughput if the workload remains memory-bound; architectural optimizations like continuous batching are essential.