Continuous Batching

Continuous batching operates at the granularity of a single forward pass (iteration), not the full sequence. The scheduler evaluates the state of all active requests after every token generation step. This allows for micro-batching decisions that are impossible in traditional serving systems. Key operational characteristics include:

• Preemption: Long-running sequences can be temporarily swapped out to prioritize latency-sensitive requests.
• Packing: Sequences of vastly different lengths coexist in the same batch without padding waste.
• Backpressure Management: The scheduler can reject or queue new requests if the current batch is computationally saturated.

A critical enabler of continuous batching is intelligent KV-Cache allocation. As sequences grow, their key-value tensors consume more GPU VRAM. Advanced implementations like PagedAttention treat the KV-Cache like virtual memory, storing it in non-contiguous blocks. This prevents memory fragmentation and allows the scheduler to physically share GPU memory between different sequences without requiring a contiguous slab for each. When a request is preempted or completed, its KV-Cache blocks are immediately freed and reassigned to a new incoming request, maximizing memory utilization alongside compute utilization.

While continuous batching dramatically increases overall system throughput (tokens generated per second), it introduces a nuanced trade-off with Time-to-First-Token (TTFT). A new request might wait in a pending queue if the current batch is full, slightly increasing its initial latency. However, this is offset by the fact that once admitted, the sequence benefits from the high utilization of the batch. System architects configure SLOs to bound this waiting time, often using a priority queue that ensures interactive requests bypass batch-processing background jobs. The net effect is a significantly lower P99 latency for the total generation time compared to static batching.

A specific evolution of continuous batching known as iteration-level scheduling or in-flight batching, pioneered by the Orca paper. This approach solves the bottleneck where a single long sequence delays the entire batch. It processes the attention computation at the token level rather than the request level. The system selects which requests participate in each forward pass of the model. This means a request generating its 500th token can be processed concurrently with a request generating its 1st token, completely decoupling batch formation from sequence length and eliminating the 'tail latency' problem inherent in request-level batching.

The primary business case for continuous batching is the direct conversion of idle GPU cycles into throughput. Static batching often results in GPU utilization as low as 30-50% due to the 'straggler' effect. Continuous batching can push utilization above 90% by ensuring the matrix multiplication units are constantly fed with a full batch dimension. This efficiency gain directly reduces the cost per token for inference providers. For enterprise deployments running on expensive A100 or H100 clusters, this optimization can represent millions of dollars in annualized infrastructure savings by reducing the total number of GPUs required to serve a given traffic load.

A memory mechanism in transformer models that stores the computed key and value tensors from previously processed tokens. During autoregressive generation, the model only computes the query, key, and value for the new token and retrieves past keys and values from the cache, avoiding redundant computation.

• Eliminates O(n²) recomputation for each new token
• Memory footprint grows linearly with sequence length
• Continuous batching relies on efficient KV-cache management to swap sequences in and out of GPU memory as they complete

The elapsed time between a user submitting a query and the language model generating the first output token. This metric directly shapes perceived interactivity and is heavily influenced by batching strategy.

• Static batching: TTFT suffers from head-of-line blocking as new requests wait for the entire batch to finish
• Continuous batching: New requests are appended immediately, dramatically reducing TTFT under load
• Target thresholds for conversational AI are typically < 200ms

A performance metric indicating the maximum response time experienced by 99% of requests, used to identify worst-case user experiences. Continuous batching directly improves P99 by preventing long-tailed sequences from blocking short ones.

• Static batching causes P99 to spike when a batch contains one long generation
• Continuous batching evicts completed sequences, allowing the straggler to continue without holding up new arrivals
• Critical for Service Level Objectives (SLOs) in production inference APIs

The percentage of a GPU's compute capacity actively performing useful work. Continuous batching maximizes this metric by filling idle cycles that would otherwise be wasted waiting for batch completion.

• Static batching: Utilization drops as sequences finish at different times, leaving bubbles of idle compute
• Continuous batching: New requests are injected into these bubbles immediately
• Typical improvement: 2-10x throughput over naive batching for mixed-length workloads

A latency-reduction technique where a small draft model predicts multiple future tokens, which a larger target model then verifies in parallel. Pairs naturally with continuous batching.

• Draft model runs autoregressively; target model checks all predictions in one forward pass
• When combined with continuous batching, the verifier batch can include tokens from multiple concurrent sequences
• Can reduce per-token latency by 2-3x without quality degradation

A flow control mechanism that signals upstream producers to slow down when a downstream system is saturated. In inference serving, backpressure interacts with batching to prevent queue bloat.

• Without backpressure, continuous batching can accumulate an unbounded request queue
• Proper backpressure signals the load balancer or client to shed load or retry with jitter
• Works alongside circuit breaker patterns to maintain system stability under traffic spikes