Dynamic Batching

Unlike static batching, which requires pre-defined batch sizes, dynamic batching continuously monitors an incoming request queue. It groups requests based on a configurable maximum batch size and a timeout window. The system waits for a short period (e.g., 50ms) to fill a batch to the target size. If the timeout is reached first, it executes the batch with whatever requests are queued, ensuring low latency even during low traffic periods. This maximizes GPU utilization without forcing clients to wait excessively.

A core challenge in batching is that requests often have variable sequence lengths (e.g., different numbers of tokens in text prompts). Dynamic batching systems handle this by:

• Padding and Masking: Sequences within a batch are padded to the length of the longest sequence in that batch. A padding mask is applied during attention computation to ignore the padded positions.
• Memory Efficiency: Advanced systems like vLLM use PagedAttention to manage memory more efficiently for variable sequences, reducing the wasted memory from padding. This allows heterogeneous requests to be processed in a single, efficient kernel launch on the GPU.

Dynamic batching is fundamentally about optimizing the throughput-latency trade-off. Key configurable parameters control this:

• Batch Timeout: A shorter timeout prioritizes low latency for individual requests but may lead to smaller, less efficient batches.
• Maximum Batch Size: A larger max size prioritizes high throughput and optimal hardware utilization but can increase latency for the first request in the batch. Serving frameworks like NVIDIA Triton Inference Server and TensorRT-LLM allow tuning these parameters per model to meet specific Service Level Objectives (SLOs).

An advanced form of dynamic batching used in autoregressive text generation. Continuous Batching (also known as iteration-level or incremental batching) addresses the inefficiency of waiting for an entire sequence to finish before starting a new one.

• Non-Blocking Execution: As soon as one request in a batch finishes generating its token, a new waiting request can be inserted into the batch for the next generation step.
• Higher GPU Utilization: This keeps the computational hardware constantly saturated, dramatically improving total tokens per second compared to static batching. It is a key feature of high-performance servers like vLLM and Text Generation Inference (TGI).

Efficient dynamic batching, especially for Large Language Models (LLMs), is tightly coupled with the management of the Key-Value (KV) Cache. This cache stores intermediate computations from previous tokens to avoid recomputation during autoregressive generation.

• Dynamic Memory Allocation: The system must dynamically allocate and deallocate portions of the KV cache as requests join, complete, or are swapped into/out of a batch.
• PagedAttention: vLLM's innovation treats the KV cache as non-contiguous, paged memory blocks. This allows flexible, fragmentation-free sharing of GPU memory across batches of variable-length sequences, which is essential for robust dynamic batching.

Dynamic batching is a foundational feature of modern ML inference platforms. Key implementations include:

• vLLM: Uses PagedAttention to enable highly efficient continuous batching for LLMs.
• NVIDIA Triton: Provides dynamic batcher as a core scheduling policy, configurable via the model's config.pbtxt.
• TensorRT-LLM: Includes an efficient in-flight batching scheduler for NVIDIA GPUs.
• TorchServe: Offers dynamic batching with configurable max batch size and timeout. These systems abstract the complexity, allowing engineers to deploy models with optimized throughput by setting a few key parameters.

Continuous batching (also known as iteration-level batching or incremental batching) is an advanced form of dynamic batching where the inference engine does not wait for an entire batch to finish before starting new requests. Instead, it continuously adds new sequences to the batch as others complete, achieving near-100% GPU utilization. This is a key feature in high-throughput servers like vLLM and NVIDIA Triton Inference Server.

• Mechanism: As soon as one sequence in the batch generates its end-of-sequence token, its slot is freed and can be immediately filled with a new pending request.
• Benefit: Eliminates the idle time inherent in static batching, where the GPU waits for the slowest sequence in the batch to finish.

Static batching is the traditional inference method where a fixed batch of requests is assembled and processed to completion before a new batch can begin. This is common in offline batch processing but is inefficient for online serving.

• Limitation: GPU utilization plummets if requests have highly variable sequence lengths, as the hardware waits for the longest sequence to finish.
• Contrast with Dynamic: Serves as the baseline that dynamic batching and continuous batching improve upon by introducing flexibility in batch composition and execution.

Kernel fusion is a low-level compiler optimization that combines multiple GPU operations (kernels) into a single, larger kernel. This reduces the overhead of launching many small kernels and improves memory bandwidth by keeping intermediate results in fast on-chip memory (registers/cache).

• Synergy with Batching: Dynamic batching creates larger, more uniform workloads that are ideal for fused kernels. Optimized inference engines like FlashAttention use kernel fusion to dramatically speed up the attention computation within transformer models, making large-batch processing more efficient.

PagedAttention is a memory management algorithm, introduced with vLLM, that treats the Key-Value (KV) cache of a transformer model like virtual memory in an operating system. It allows non-contiguous storage of the cache in fixed-size blocks (pages).

• Enabler for Dynamic Batching: This is the foundational innovation that makes efficient dynamic batching possible for variable-length sequences. By eliminating internal memory fragmentation, it allows the system to flexibly allocate cache for incoming requests of any size, which is essential for high-throughput serving with unpredictable workloads.

Model parallelism is a strategy for distributing a single large neural network across multiple GPUs or other accelerators. This is necessary when a model's weights or activations are too large to fit on one device's memory.

• Interaction with Batching: Dynamic batching operates at a higher level, optimizing request scheduling within a parallelized model setup. Serving systems must coordinate batching logic across all devices in the model-parallel group, adding complexity to scheduling decisions to avoid stragglers that slow down the entire pipeline.

Request scheduling is the overarching system component that decides the order and grouping of incoming inference requests. Dynamic batching is a specific scheduling policy within this component.

• Policies: Schedulers implement algorithms like First-Come-First-Served (FCFS), Shortest-Job-First (SJF), or more complex quality-of-service (QoS) aware policies.
• Trade-offs: The scheduler must balance competing goals: maximizing throughput (requests/second), minimizing latency (time to first token, time per output token), and meeting service-level agreements (SLAs). Dynamic batching is primarily a throughput-maximizing technique.