Dynamic Batching

Unlike static batching, dynamic batching groups requests based on their sequence length and arrival time. The server uses a batching window or timeout to collect incoming requests. It then forms batches where sequences are padded to the length of the longest sequence in the batch, minimizing wasted computation. This is critical for language models where input prompts vary drastically in size.

• Example: A server with a 50ms window might batch a 10-token query with a 45-token query, padding the shorter one to 45 tokens for parallel processing.

The primary goal is to keep GPU/TPU compute units saturated. By dynamically forming larger batches, the server amortizes the fixed overhead of launching a kernel across more work items. This transforms sporadic, single requests into dense tensor operations, which modern accelerators are designed to execute with extreme parallelism. The throughput gains are most significant when request rates are high but individually would not fill the hardware's compute capacity.

Dynamic batching introduces a fundamental trade-off. The batching delay (time spent waiting for requests to form a batch) increases latency for individual requests but boosts overall system throughput (requests/second). Servers provide knobs to control this:

• Maximum Batch Size: Hard limit to prevent out-of-memory errors.
• Batching Timeout: Maximum wait time for the first request in a queue before executing the batch, preventing excessive latency.

Tuning these parameters is essential for meeting specific Service Level Objectives (SLOs).

For autoregressive text generation, basic dynamic batching is insufficient because sequences within a batch generate tokens at different rates. Continuous batching (or iterative batching) is an advanced extension. It allows new requests to join a running batch as soon as previous requests finish generation, rather than waiting for the entire batch to complete. This technique, used by servers like vLLM and TGI, decouples latency from the slowest request in the batch and can improve GPU utilization to over 70% for generative tasks.

Dynamic batching must efficiently handle the variable memory footprint of different batch compositions. Advanced inference servers like NVIDIA Triton use ragged batching or similar techniques to minimize padding overhead. For generative models, managing the Key-Value (KV) Cache per request within a dynamic batch is complex. Engines like vLLM implement PagedAttention, which manages the KV cache in non-contiguous, paged blocks, allowing for efficient memory sharing and fragmentation avoidance as requests dynamically enter and exit the batch.

A robust scheduling algorithm is required to manage incoming requests. Servers typically maintain one or more priority queues. The scheduler decides:

• When to create a new batch from queued requests.
• How to prioritize requests (e.g., FIFO vs. based on sequence length).
• How to handle priority inference requests that may skip the queue.

This scheduler works in tandem with the batching timeout to ensure system responsiveness under load.

Also known as iterative batching, this is an advanced form of dynamic batching designed for autoregressive text generation. Instead of waiting for an entire batch to finish, the server adds new requests to a running batch as soon as slots become available from completed sequences. This maintains high GPU utilization and throughput even when requests have highly variable output lengths.

• Key Mechanism: Manages a pool of active sequences, scheduling computation only for the tokens that are ready to be generated next.
• Contrast with Static Batching: Eliminates the idle time inherent in static batches where fast requests wait for slower ones.
• Primary Benefit: Enables high-throughput serving of LLMs with predictable latency for streaming responses.

A specialized software system that hosts machine learning models and serves predictions via network APIs (e.g., HTTP/gRPC). It is the foundational platform that implements dynamic batching.

Core responsibilities include:

• Model Lifecycle Management: Loading, unloading, and versioning of models.
• Request Orchestration: Queuing, scheduling, and forming batches like dynamic batching.
• Hardware Acceleration: Optimizing execution for GPUs, NPUs, or CPUs.
• API Exposure: Providing standardized endpoints for client applications.

Examples include NVIDIA Triton Inference Server, vLLM, and Hugging Face's Text Generation Inference (TGI).

vLLM is an open-source, high-throughput LLM serving engine. Its performance is largely due to PagedAttention, an innovative algorithm for managing the Key-Value (KV) Cache.

• The Problem: The KV cache for long sequences consumes contiguous, variable-sized memory blocks, leading to fragmentation and wasted GPU memory.
• PagedAttention's Solution: It borrows the concept of virtual memory and paging from operating systems. The KV cache is divided into fixed-size blocks that can be non-contiguously stored in GPU memory.
• Impact on Batching: This efficient memory management allows vLLM to support much larger batch sizes and longer context lengths than naive implementations, making dynamic and continuous batching far more effective.

An inference architecture where a single base model instance can dynamically load and switch between multiple trained adapter or LoRA modules. This is critical for cost-effective deployment of many fine-tuned variants.

• Architecture: The large base model weights remain frozen in GPU memory. Small adapter weights are stored in host memory or SSD and swapped in/out per request.
• Adapter Switching: Routing logic (based on a request header like task-id) selects the correct adapter set before executing the dynamic batch.
• Benefit: Enables multi-tenancy and personalized models without the cost of loading N full model copies. Dynamic batching can occur across requests destined for different adapters on the same base model.

These terms describe the initialization state of a served model and directly impact the effectiveness of dynamic batching.

• Cold Start: The high-latency period when a model endpoint must be scaled from zero or first deployed. The model is not in memory, requiring time to load weights, compile kernels, and initialize. Dynamic batching cannot begin until this process completes.
• Model Warm-up: A proactive process to eliminate cold start latency for production traffic. It involves:
  • Pre-loading the model into GPU memory after deployment.
  • Executing a series of dummy inference requests with typical batch sizes.
  • This triggers kernel compilation and populates caches, ensuring the first real user request meets latency SLOs and can immediately benefit from dynamic batching.

Autoscaling is the cloud infrastructure counterpart to application-level optimizations like dynamic batching. The Horizontal Pod Autoscaler (HPA) in Kubernetes automatically adjusts the number of inference server pods based on demand.

• How it Works: The HPA monitors metrics like average CPU utilization, memory consumption, or custom metrics (e.g., request queue length). If a threshold is exceeded, it spins up new pods.
• Synergy with Dynamic Batching: Dynamic batching maximizes throughput within a pod. Autoscaling adjusts the number of pods to handle the total request load. They work together to control cost and performance.
• Key Metric: Queue length is often the best signal for scaling inference workloads, indicating that incoming requests are waiting for batch formation and processing.