Continuous Batching

Unlike static batching which waits for a full batch to form, continuous batching treats the batch as a dynamic set. New requests are injected into the running batch as soon as a GPU kernel slot becomes available from a completed request. This eliminates idle time where the accelerator waits for new work, a state known as bubble underutilization. The scheduler operates at the granularity of a single generation step, making decisions on a per-token basis.

The primary technical benefit is driving GPU utilization towards its theoretical maximum, often exceeding 80-90% during sustained load. By keeping the computational units (SMs on NVIDIA GPUs) constantly saturated with work, it directly translates capital expenditure on hardware into higher throughput (requests per second). This is critical for cost-effective serving, as idle GPU time is the single largest contributor to wasted inference budget.

Continuous batching provides lower per-request latency, especially for shorter generations, by minimizing queue time. Requests are not held waiting for a large batch to assemble. In systems like vLLM, this is combined with PagedAttention to manage memory efficiently, further reducing delays caused by memory fragmentation. The result is a more responsive user experience and the ability to meet stricter latency SLAs (e.g., p99 < 1 second).

It requires sophisticated KV cache management. As requests finish generation at different times, their allocated cache memory must be instantly reclaimed for new requests. Systems implement paged memory for the KV cache, analogous to virtual memory in operating systems. This allows non-contiguous, fine-grained allocation and deallocation, preventing memory waste and enabling the serving of more concurrent requests within fixed GPU memory constraints.

It is perfectly suited for the unpredictable nature of autoregressive text generation, where output length varies per request (e.g., a 10-token code completion vs. a 500-token email draft). The scheduler handles this heterogeneity seamlessly. This contrasts with tasks like image classification, where all inputs have fixed computational cost, making traditional batching sufficient.

Continuous batching is the enabling technology behind cost-effective multi-tenant serving platforms. It allows a single GPU cluster to efficiently interleave requests for different models, tasks, or customers. When combined with multi-adapter inference for PEFT models, the scheduler can dynamically load different LoRA adapters for individual requests within the same batch, all while maintaining high base model weight reuse.

At its core, continuous batching replaces static batching (where a fixed batch is processed until all sequences are complete) with a dynamic scheduler. The algorithm operates at the iteration level (per generated token).

• Scheduler: Maintains a pool of active requests. At each decoding iteration, it selects all requests that are ready for their next token generation.
• KV Cache Management: Newly scheduled requests allocate KV cache; completed requests free their cache immediately.
• Execution: The engine constructs a new, potentially different-sized batch tensor for each forward pass, containing the next token's context for all active requests.
• This ensures the GPU is never idle waiting for a long sequence to finish when shorter ones are complete, maximizing hardware utilization and throughput.

In production, continuous batching engines are rarely standalone. They are orchestrated within larger systems to manage scale, cost, and multiple model types.

• Kubernetes Operators: Specialized controllers (like Kserve, KServe's ModelMesh, or Ray Serve) automate the deployment, scaling, and management of inference engines across clusters.
• Multi-Model Serving: Platforms like Triton or Ray Serve can load balance traffic and co-locate multiple different models (e.g., an embedding model, a classifier, and an LLM) on the same GPU cluster, using continuous batching for eligible models.
• Queue Management: A front-end request queue (often with priority scheduling) feeds the continuous batching engine, ensuring fair access and managing traffic bursts.

Implementing continuous batching involves measurable trade-offs between key serving metrics.

• Primary Benefit: Increased Throughput. By keeping the GPU saturated, requests per second (RPS) can increase by 5-10x compared to static batching.
• Trade-off: Tail Latency. While average latency improves, the p99 latency (the slowest 1% of requests) can increase. This is because a new, fast request can be briefly delayed waiting for the current batch iteration to complete.
• Optimal Batch Size: The scheduler dynamically adjusts the batch size. An engine's efficiency is measured by its time to first token (TTFT) and time per output token (TPOT) across varying loads.
• The goal is to maximize total goodput (successful tokens per second) while meeting latency Service Level Agreements (SLAs).

Dynamic batching is a foundational inference optimization where an inference server groups multiple incoming requests into a single batch for parallel processing on hardware accelerators like GPUs. Unlike static batching, the batch size and composition can change per inference cycle.

• Key Mechanism: The server waits for a short time window to accumulate requests, then executes them as one batch, amortizing the fixed cost of loading the model onto the GPU across many inputs.
• Trade-off: Introduces a slight latency penalty (wait time) for significantly improved throughput and GPU utilization.
• Contrast with Continuous Batching: Dynamic batching typically processes the entire batch to completion before starting a new one, whereas continuous batching allows requests to join and exit a running batch at the token level.

Throughput in model serving is the number of inference requests a system can process successfully per unit of time, typically measured in requests per second (RPS) or tokens per second.

• Primary Goal of Continuous Batching: Maximize throughput by ensuring the GPU is never idle. By dynamically filling "bubbles" left by completed sequences, it sustains near 100% hardware utilization.
• Trade-off with Latency: While continuous batching optimizes for aggregate throughput, individual request latency can be variable depending on when it enters the batch and its generation length.
• Measurement: Critical for calculating cost per inference and capacity planning. High throughput directly reduces the infrastructure cost per request.

Online inference (or real-time inference) is a serving pattern where predictions are generated synchronously and returned with low latency in response to individual user requests, typically via an API.

• Continuous Batching Context: This is the operational mode for which continuous batching is designed. It enables online inference for LLMs—which are inherently sequential and slow—to be viable at scale by improving throughput without sacrificing interactive latency guarantees.
• Contrast with Batch Inference: Batch inference processes large, accumulated datasets asynchronously, prioritizing total job completion time over individual request latency. Continuous batching brings some throughput benefits of batch processing to the online setting.

Multi-adapter inference is a serving architecture for models fine-tuned with Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA. It allows a single base model to dynamically load and execute different lightweight adapter modules per request.

• Synergy with Continuous Batching: A serving system can maintain a continuously batched execution engine for the frozen base model while rapidly swapping adapter weights in and out for different requests. This enables efficient multi-tenant or multi-task serving from a single GPU.
• Runtime Adapter Injection: The technique of dynamically merging adapter weights with the base model at inference time. This avoids the need to store and load hundreds of full, multi-gigabyte models.