Dynamic Batching

The core mechanism of dynamic batching is its ability to accept incoming inference requests of different sizes and sequence lengths, grouping them into a single computational batch for parallel processing. This is critical for serving language models where user prompts vary dramatically in token count. The system typically uses padding or masking to handle the irregular tensor shapes within the batch. This aggregation happens in a holding queue, with the batch being dispatched for execution once a configured threshold—such as maximum batch size, time delay, or total token count—is met.

Dynamic batching explicitly manages the fundamental trade-off between throughput (requests processed per second) and latency (time to first token). Key configurable parameters control this balance:

• Maximum Batch Size: Limits how many requests are grouped to prevent excessive latency for the first request in the queue.
• Timeout Window: Sets a maximum wait time for the queue to fill before executing the current batch, ensuring requests are not delayed indefinitely.
• Token-Based Batching: Groups requests based on total tokens rather than request count, leading to more consistent GPU utilization. The system continuously optimizes this trade-off based on real-time load.

For autoregressive text generation, a specialized form called continuous batching (or iteration-level scheduling) is used. In this scheme, a batch of sequences is processed in parallel. When some sequences finish generating their end-of-sequence token, they are removed from the batch, and new waiting sequences are immediately inserted into the vacant slots without stopping the GPU. This creates a 'rolling' batch that maintains near-100% GPU utilization throughout the generation process, a significant improvement over static batching where the GPU would idle between generations.

Efficient dynamic batching requires sophisticated memory management for the Key-Value (KV) cache in Transformer models. As sequences of variable lengths are added and removed from the rolling batch, their KV caches must be allocated and freed without fragmentation. Advanced systems like vLLM employ PagedAttention, which manages the KV cache in non-contiguous, paged memory blocks (similar to virtual memory in operating systems). This allows for efficient sharing of memory between sequences and eliminates wasted space from padding, dramatically improving memory utilization for large language models.

The primary goal is to saturate the compute units of the target hardware (GPU, NPU). By continuously feeding grouped operations, dynamic batching keeps Tensor Cores or Matrix Multiply Units busy, amortizing the fixed overhead of kernel launches and memory transfers across many requests. This is especially effective for smaller requests that individually would not fully utilize the hardware's parallel capabilities. The increase in throughput is often non-linear; doubling the batch size can more than double the tokens/sec processed, up to the hardware's memory bandwidth or compute limit.

Dynamic batching is a core feature of high-performance inference servers and runtimes. Key implementations include:

• NVIDIA Triton Inference Server: Uses a dynamic batcher scheduler with configurable queue policies.
• vLLM: Implements continuous batching with PagedAttention for LLM serving.
• TensorRT LLM: Provides in-flight batching capabilities for NVIDIA GPUs.
• ONNX Runtime: Supports dynamic batching through its execution providers. These systems expose APIs and configuration parameters (e.g., preferred_batch_size, max_queue_delay_microseconds) to tune batching behavior for specific deployment scenarios.

These are newer, high-performance frameworks with advanced batching and execution optimizations.

• SGLang: A co-design of frontend language and backend runtime for LLMs. It uses RadixAttention, a technique that automatically caches and reuses the KV cache of common prefixes across different requests (e.g., shared system prompts), dramatically improving throughput for complex prompts.
• LMDeploy: A toolkit from the InternLM team for compressing, serving, and deploying LLMs. Its Turbomind backend features a persistent batch scheduler, efficient KV cache management, and continuous batching. It supports W4A16 (4-bit weight, 16-bit activation) quantization and NVIDIA TensorCore-accelerated INT4 inference.

Static batching is an inference strategy where a fixed number of input samples are grouped into a batch before processing. Unlike dynamic batching, the batch size is predetermined and constant throughout execution.

• Key Mechanism: All requests in the batch must be ready before inference begins. The system waits for the batch to fill, which can increase latency for early-arriving requests.
• Primary Use Case: Best suited for offline processing or scenarios with predictable, uniform request rates where maximizing throughput is the sole objective and latency variance is acceptable.
• Trade-off: Provides high, predictable hardware utilization but poor latency characteristics for real-time, interactive applications.

Continuous batching (or iteration-level scheduling) is an advanced optimization for autoregressive models like LLMs. It dynamically manages requests within a batch at the granularity of token generation steps.

• Key Mechanism: New requests can be added to a running batch, and finished requests can be removed, without waiting for the entire batch to complete. This is implemented in systems like vLLM and NVIDIA's Triton Inference Server.
• Primary Use Case: Critical for serving large language models where generating a single output involves many sequential steps. It dramatically improves GPU utilization and throughput for chat and completion endpoints.
• Relation to Dynamic Batching: Can be seen as a specialized, fine-grained form of dynamic batching applied to the sequential decoding process.

Inference latency is the time delay between submitting an input to a model and receiving its output. It is a critical performance metric for real-time applications.

• Measurement: Typically reported in milliseconds (ms) as p50, p95, and p99 percentiles to understand tail latency.
• Factors: Influenced by batch size, model architecture, hardware (CPU/GPU/NPU), data transfer times, and optimization techniques like batching. Dynamic batching explicitly trades off some latency for improved throughput.
• Targets: Requirements vary from sub-10ms for autonomous vehicles to several hundred milliseconds for conversational AI.

Hardware utilization measures how effectively computational resources (e.g., GPU cores, NPU compute arrays) are kept busy during model execution. High utilization is key to cost-effective inference.

• Metric: Often measured as a percentage of peak theoretical FLOPS (Floating Point Operations Per Second) or via hardware profiling tools like NVIDIA's Nsight Systems.
• The Batching Link: Small batch sizes often lead to low utilization due to kernel launch overhead and memory bandwidth underutilization. Dynamic batching aims to aggregate work to keep hardware saturated.
• The Balance: The goal is to maximize utilization without causing excessive queuing delays that violate latency service-level agreements (SLAs).

A compute graph is a directed acyclic graph (DAG) representation of a neural network, where nodes are operations (ops) and edges are data tensors. It is the fundamental structure optimized by inference engines.

• Role in Optimization: Frameworks like TensorRT, ONNX Runtime, and TVM perform transformations on the compute graph to enable batching and other optimizations (e.g., operator fusion, constant folding).
• Batching Support: The graph must be defined with batch dimensions (often symbolic) to allow runtime binding of different batch sizes, which is a prerequisite for dynamic batching.
• Execution: The optimized graph is executed by a runtime scheduler, which handles the dynamic allocation of batches to computational kernels.

Kernel optimization involves crafting highly efficient, low-level code for fundamental operations (kernels) like matrix multiplication (GEMM) and convolution on specific hardware.

• Objective: Minimize execution time by leveraging hardware features such as vectorization, tensor cores, and memory hierarchy (registers, cache, RAM).
• Synergy with Batching: Optimized kernels are essential to realize the performance gains from larger batch sizes. Poorly optimized kernels can negate the benefits of batching.
• Methods: Includes manual tuning, auto-tuning (as in AutoTVM), and using vendor-optimized libraries like cuBLAS (NVIDIA) or oneDNN (Intel).