Continuous Batching

Unlike static batching, which waits for a fixed batch size to accumulate or a fixed time window to expire, continuous batching adds new requests to an in-flight batch as soon as GPU resources become available from completed sequences. This eliminates idle time and keeps the GPU saturated, dramatically improving throughput (Queries Per Second) under variable or low request loads.

• Key Mechanism: A central scheduler monitors the generation state of all active requests.
• Benefit: New requests do not wait for a full batch; they start execution almost immediately, reducing average latency.

The scheduler operates at the granularity of a single decoding iteration (the generation of one token), not the entire request. When one sequence within a batch finishes generation (hits an end-of-sequence token), its slot in the batch is immediately freed. The scheduler can then insert a prefill computation for a new waiting request into that slot for the next iteration.

• Contrast: Traditional batching processes all sequences for their full length together.
• Result: Efficient handling of requests with highly variable output lengths, common in conversational AI.

Continuous batching is often paired with PagedAttention, an algorithm for managing the Key-Value (KV) Cache. In variable-length continuous batching, memory for the KV cache becomes fragmented. PagedAttention treats the KV cache as virtual memory, dividing it into fixed-size blocks. This allows:

• Non-contiguous, paged storage of the KV cache for each sequence.
• Elimination of memory waste from internal fragmentation.
• Safe and efficient memory sharing for identical prompts in different requests.

This combination is foundational to engines like vLLM, enabling high throughput with many concurrent requests.

By reducing request queuing delay, continuous batching directly improves tail latency metrics (P95, P99). In static batching, a request arriving just after a batch starts must wait for the entire batch to complete, potentially causing a long delay. With continuous batching, the wait time is bounded by the time to generate a single token from the longest-running sequence in the current batch, which is typically much shorter.

• Impact: Provides more consistent and predictable latency for end-users.
• Use Case: Critical for interactive applications like chatbots where perceived responsiveness is key.

Continuous batching inherently accommodates requests with different input (prompt) and output (completion) lengths. The scheduler independently tracks the progress of each sequence. This is a major advantage over static batching, which requires padding all sequences to the length of the longest one in the batch, leading to significant computational waste on padding tokens.

• Efficiency Gain: No compute is wasted on padding.
• Real-world Fit: Perfectly suited for production traffic where prompt and response lengths are highly variable.

Continuous batching allows operators to tune the system's position on the throughput-latency curve. By adjusting the maximum number of concurrent requests the batch can hold, you can prioritize higher throughput (more concurrent requests) or lower latency (fewer concurrent requests).

• High Throughput Mode: Maximize GPU utilization and QPS for batch processing or high-load scenarios.
• Low Latency Mode: Minimize time-to-first-token for interactive applications.
• Dynamic Adjustment: This parameter can be modified based on real-time traffic patterns.

The total time delay between submitting an input to a machine learning model and receiving its output. This is the overarching performance metric that continuous batching directly aims to reduce by maximizing hardware utilization. It encompasses:

• Compute time on the GPU/CPU
• Memory access and data transfer
• Queuing delay before execution begins Continuous batching attacks the queuing and compute components by keeping the processor constantly occupied.

The traditional batching method where a fixed set of requests are collected, processed simultaneously, and all results are returned at the same time. This contrasts with continuous batching.

Key drawbacks:

• Low utilization: The entire batch waits for the slowest request to finish.
• High tail latency: P99 latency suffers as short requests wait behind long ones.
• Inefficient for streaming: Cannot stream tokens as they are generated. Continuous batching was developed to solve these fundamental inefficiencies in static batching.

A memory management algorithm for the Key-Value (KV) cache in attention mechanisms, introduced by the vLLM serving engine. It is a critical enabling technology for continuous batching.

It treats the KV cache like virtual memory, using blocks or 'pages' that can be non-contiguously allocated. This allows:

• Efficient memory reuse for finished sequences.
• Dynamic addition of new sequences to a running batch.
• Elimination of memory fragmentation, which is a major bottleneck for naive continuous batching implementations.

A graph that plots the relationship between a system's request throughput (e.g., Queries Per Second) and its corresponding average or tail latency. It defines the operational trade-off for any serving system.

Continuous batching's impact: It shifts this curve favorably, enabling higher throughput at the same latency or lower latency at the same throughput compared to static batching. The goal is to find the 'knee' of the curve—the optimal point before latency increases exponentially.

A core throughput metric measuring the number of inference requests a system can successfully process each second. It is the primary beneficiary of continuous batching optimization.

How continuous batching improves QPS: By eliminating idle GPU cycles that occur in static batching when:

• Waiting to fill a batch.
• Fast requests waiting for slow ones.
• The KV cache is inefficiently managed. The technique maximizes useful computation per second, directly raising the sustainable QPS for a given hardware footprint and latency Service Level Objective (SLO).