Continuous Batching

Unlike static batching, which waits for a fixed batch size or time window, continuous batching processes requests as they arrive. The scheduler dynamically adds new requests to the current batch as soon as previous requests finish generating tokens. This eliminates idle time where GPUs would wait for a full batch, dramatically improving throughput and reducing tail latency for individual users. It's the computational equivalent of a just-in-time assembly line for model inference.

The fundamental unit of execution is the iteration, where the model generates one token for all active requests in parallel. Within each iteration:

• Active Sequences: Requests that are still generating tokens are processed.
• Finished Sequences: Requests that have produced an end-of-sequence token are removed from the batch.
• New Sequences: Incoming requests are added to the batch. This fine-grained control allows the system to maintain a consistently high GPU utilization by always having work to do, rather than batching at the coarse request level.

Continuous batching's efficiency is enabled by sophisticated KV Cache management. Systems like vLLM implement PagedAttention, which treats the KV cache like virtual memory:

• Non-Contiguous Memory: The KV cache for each sequence is stored in non-contiguous blocks, similar to pages in an OS.
• Efficient Sharing: For prompts with common prefixes (e.g., system prompts), the KV cache pages can be shared across requests, saving significant memory.
• Elimination of Fragmentation: This paging approach prevents memory fragmentation caused by variable-length sequences finishing at different times, allowing for near-100% memory utilization.

The primary performance benefit is a massive increase in tokens per second served per GPU, especially under real-world, variable request patterns. Benchmarks show throughput improvements of 5x to 24x over static batching for popular models. This is because the GPU is almost never idle. The system achieves high throughput not just at peak load but also during periods of lower, sporadic traffic, making it cost-effective for production deployments where request rates are unpredictable.

By immediately adding requests to an ongoing batch, continuous batching minimizes queue time. A user's request doesn't wait for a batch timer to expire or for n other requests to arrive. This significantly improves the time to first token (TTFT). While the total generation time for a long output is constrained by the model's speed, the perceived responsiveness is much better, leading to a superior user experience in interactive applications like chatbots.

Continuous batching is a core feature of modern, high-performance LLM serving engines. Key implementations include:

• vLLM: Uses PagedAttention as its foundation for continuous batching.
• Text Generation Inference (TGI): Hugging Face's toolkit featuring continuous batching (called "dynamic batching").
• TensorRT-LLM: NVIDIA's SDK includes continuous batching optimizations for GPUs.
• Triton Inference Server: Supports dynamic batching for ensembles, which can be configured for LLM-like behavior. These engines abstract the complexity, allowing developers to benefit from continuous batching via standard APIs.

The PagedAttention algorithm is the foundational memory optimization that makes practical continuous batching possible. It solves the problem of KV cache fragmentation by:

• Dividing the cache into fixed-size blocks (pages).
• Allocating blocks to requests non-contiguously, as needed.
• Maintaining a block table for each request sequence, similar to an operating system's page table. This allows the system to efficiently serve hundreds of concurrent requests with varying sequence lengths, which is impossible with static, contiguous cache allocation.

Implementing continuous batching directly targets key production metrics:

• Throughput: Increases the total number of tokens generated per second by keeping the GPU constantly occupied, often by 5-10x compared to static batching.
• Latency: Reduces queue time for individual requests, as they don't wait for a full static batch to form. However, the time-to-first-token (TTFT) for a request may be slightly higher if it must wait for the current batch's forward pass to complete.
• GPU Utilization: Drives utilization toward 100%, eliminating the idle time inherent in static batching where GPUs wait for new requests to fill a batch.

Dynamic batching is a predecessor to continuous batching where a serving system waits for a short period to collect multiple incoming requests, groups them into a batch, and processes them together in a single forward pass. Unlike continuous batching, it typically requires all requests in the batch to finish generation before the next batch can start, leading to inefficient GPU utilization when requests have varying output lengths.

• Key Limitation: Known as the straggler problem, where a single long-running request blocks the entire batch.
• Use Case: Effective for tasks with uniform, predictable output lengths, such as classification or embedding generation.

Iteration-level scheduling is the underlying scheduling paradigm that enables continuous batching. Instead of scheduling at the request level, the system schedules computation at the level of a single generation step (iteration) for each sequence. This allows the inference engine to:

• Dynamically Add new requests to the active batch as soon as previous requests finish.
• Remove completed sequences from the batch without stopping execution.
• Maximize GPU Utilization by ensuring the processing tensor is always fully packed with active tokens.

This fine-grained control is what transforms static batches into a continuous, fluid computation stream.

Token streaming (or response streaming) is a user-facing feature often implemented alongside continuous batching. It delivers generated tokens to the client application as soon as they are produced, rather than waiting for the entire sequence to complete.

• User Experience: Provides low perceived latency, as users see text appear incrementally.
• System Synergy: Works efficiently with continuous batching because the engine is already generating tokens for multiple sequences in an interleaved, iteration-by-iteration fashion. Streaming simply exposes this incremental progress via the API (e.g., using Server-Sent Events).
• Contrast with Dynamic Batching: Traditional dynamic batching often cannot stream effectively, as the entire batch must finish before any output is released.

In-flight batching is a synonym often used interchangeably with continuous batching. It emphasizes the core characteristic that the composition of the batch is not static but changes 'in-flight' during the execution of a forward pass.

• Key Insight: The batch is a living set of active sequences. This requires sophisticated attention masking to ensure each sequence only attends to its own prior tokens and padding management to handle variable numbers of active tokens per iteration.
• Implementation Challenge: Requires deep integration into the model's forward pass, as implemented in engines like vLLM, TGI, and TensorRT-LLM.