Dynamic Batching

The inference server maintains a request queue where incoming queries are held for a configurable maximum batching delay (e.g., 50ms). The batching scheduler then groups requests from this queue into a single batch tensor. Key parameters include:

• Batch Size: The maximum number of requests grouped together, limited by GPU memory.
• Timeout: The maximum time a request waits in the queue before being processed, balancing latency and throughput. This contrasts with static batching, which requires a fixed batch size and waits to fill it, often increasing tail latency.

The primary goal is to saturate the parallel compute units of hardware accelerators like GPUs or NPUs. By forming larger, contiguous batches, dynamic batching:

• Amortizes overhead costs of kernel launches and memory transfers across multiple requests.
• Increases compute density, allowing the GPU's Streaming Multiprocessors (SMs) to operate more efficiently.
• Maximizes throughput, often achieving a 5x to 10x improvement in requests per second (RPS) compared to sequential processing, especially for smaller models where individual inferences underutilize the hardware.

A core challenge is that requests (especially for language models) have variable input sequence lengths. To form a rectangular batch tensor, the system must pad shorter sequences to match the longest sequence in the batch. This creates computational waste on padded tokens. Advanced implementations use:

• Bucket-based batching: Grouping requests with similar sequence lengths into buckets to minimize padding.
• Attention mask: A binary mask that tells the model to ignore computations on padded positions, preventing them from affecting the output.

Dynamic batching is often used for encoder-based or non-autoregressive models where all outputs are generated simultaneously. Continuous batching (or iteration-level batching) is a more advanced variant optimized for autoregressive generation in LLMs (like GPT). In continuous batching:

• New requests are added to a running batch as soon as previous requests finish generating their current token.
• It employs sophisticated memory management like PagedAttention (used in vLLM) to handle variable sequence lengths efficiently.
• This eliminates the need for padding within the generation loop, achieving near-optimal GPU utilization for text generation.

Dynamic batching is crucial for cost-effective serving of Parameter-Efficient Fine-Tuned (PEFT) models. In a multi-adapter inference architecture:

• A single base model instance serves multiple tasks, each defined by a different adapter module (e.g., LoRA weights).
• The batching scheduler must group requests not only by sequence length but also by adapter ID.
• Runtime adapter injection allows the system to dynamically load the correct adapter weights for each batch, enabling efficient multi-tenant or multi-task serving from a shared GPU resource.

The process of transforming a model from a framework-specific format (e.g., PyTorch, TensorFlow) into a highly optimized, hardware-specific executable. This step is often a prerequisite for achieving peak performance with batching techniques.

• Purpose: Minimizes inference latency and maximizes throughput by leveraging hardware-specific kernels and operator fusion.
• Common Formats: ONNX Runtime, TensorRT, OpenVINO, and AWS Neuron.
• Relation to Batching: Compiled models often include optimized kernels for batched matrix operations, making dynamic batching more efficient.

A serving architecture that enables a single base model to dynamically load different lightweight PEFT modules (e.g., LoRA adapters) per request. This is a key deployment pattern for models fine-tuned with parameter-efficient methods.

• Core Concept: Decouples the large, frozen base model from small, task-specific adapter weights.
• Efficiency: Allows multi-tenant or multi-task serving without storing dozens of full model copies.
• Runtime Process: Uses runtime adapter injection to merge adapter weights with the base model's weights on-the-fly during inference.

The dynamic, automated adjustment of compute resources (e.g., Kubernetes pods, cloud instances) for a model serving deployment based on real-time demand. It works in concert with batching to manage cost and performance.

• Trigger Metrics: Scales based on CPU/GPU utilization, memory pressure, or request queue length.
• Interaction with Batching: Effective autoscaling policies must account for the increased memory and compute requirements of larger batch sizes to prevent resource exhaustion during traffic spikes.

The two fundamental serving patterns that define latency and throughput requirements, guiding when to use dynamic batching.

• Online Inference: Synchronous, low-latency requests (e.g., chatbot responses). Dynamic batching is critical here to improve throughput while meeting latency SLAs.
• Batch Inference: Asynchronous, high-throughput processing of accumulated data (e.g., daily scoring). Uses large, static batches where latency is not a primary constraint.
• Async Inference: A hybrid pattern where requests are queued and processed in batches, with clients polling for results, suitable for long-running operations.

The practice of deploying models that have been quantized (reduced precision from FP32 to INT8/FP16) to decrease memory footprint and accelerate computation. This directly enhances the efficiency of batched execution.

• Impact on Batching: Lower precision models consume less memory per instance, allowing for larger batch sizes on the same hardware, which improves throughput.
• Serving Runtimes: Specialized engines like TensorRT and ONNX Runtime execute quantized models with optimized kernels that are highly efficient for batched inputs.