Static Batching

Once a batch is formed, its composition is locked for the entire duration of processing. All requests in the batch start and finish together. This is in contrast to dynamic or continuous batching, where requests can join or exit a batch at different times. The scheduler typically forms a batch based on a fixed batch size or after waiting for a predetermined batch window timeout.

The system processes the entire batch in a synchronous, barrier-like fashion. The forward pass for all sequences in the batch is executed simultaneously. No single request within the batch receives its output until the slowest request in that batch has finished its generation. This leads to predictable, but often suboptimal, latency profiles.

Static batching maximizes hardware utilization and throughput when processing large volumes of requests with similar characteristics. By fully saturating the GPU's parallel compute units with a large, uniform batch, it amortizes the fixed costs of kernel launch overhead and memory transfers. This makes it efficient for offline or batch processing jobs.

• Ideal for: Model evaluation, bulk embedding generation, offline summarization.
• Inefficient for: Interactive chat with highly variable prompt lengths.

A major drawback is susceptibility to head-of-line blocking. A single long-running request (e.g., a very long output generation) stalls the entire batch, delaying all other requests. This directly inflates tail latency metrics (p95, p99), making it unsuitable for latency-sensitive applications. Padding is often required to align sequence lengths, wasting compute on padding tokens.

The scheduling logic is straightforward. An inference server implementing static batching typically uses a request queue and a simple policy: 'wait for N requests or T seconds, then process.' There is no need for complex iteration-level scheduling or state tracking for individual requests mid-batch. This simplicity reduces system complexity and debugging overhead.

Static batching is the baseline against which advanced techniques are compared. Continuous batching (or iterative batching) eliminates its core limitations by:

• Dynamically adding new requests to a running batch.
• Releasing finished sequences immediately, freeing resources.
• Eliminating head-of-line blocking for variable-length generations. This shift moves the optimization goal from batch-level throughput to request-level latency while maintaining high GPU utilization.

Static batching is the standard method for offline batch processing, where a large, pre-defined dataset is processed in fixed-size chunks. This is common in:

• Model evaluation on validation/test sets.
• Embedding generation for populating a vector database.
• Content moderation scans of pre-uploaded media.
• Data preprocessing where a model is used as a feature extractor.

The fixed batch size allows for precise resource provisioning and predictable job completion times, as the entire workload is known in advance.

In analytics and reporting workloads, static batching efficiently processes aggregated queries. Examples include:

• Sentiment analysis on a day's worth of social media posts.
• Batch forecasting for inventory or demand predictions run on a scheduled basis (e.g., nightly).
• Anomaly detection on log files collected over a fixed period. These applications prioritize aggregate throughput over individual request latency. The system ingests a known volume of data, forms optimally sized batches for the hardware, and processes them sequentially to maximize GPU utilization.

While training uses dynamic batching, related ancillary tasks often rely on static batches:

• Generating synthetic training data using a large generative model.
• Running inference for teacher labels in knowledge distillation pipelines.
• Pre-computing features for classical machine learning models. These are embarrassingly parallel tasks with no inter-request dependencies. Static batching provides a simple, fault-tolerant execution model where the failure of one batch does not cascade, and the job can be resumed from the last completed batch.

Early model serving frameworks and many first-generation inference servers exclusively used static batching due to its implementation simplicity. It remains in use for:

• Legacy applications where refactoring for dynamic batching is costly.
• Research prototypes where serving complexity is a distraction.
• CPU-based inference on large, homogeneous datasets where the overhead of dynamic scheduling may outweigh its benefits. The synchronization point at batch completion simplifies state management and result aggregation, making debugging and reasoning about system behavior more straightforward.

This card defines the core constraint of static batching. It maximizes system throughput (tokens/second) by fully utilizing parallel compute during the prefill phase and keeping the GPU busy.

However, it introduces significant latency penalties:

• Queueing Delay: A request must wait for the batch to fill.
• Head-of-Line Blocking: A single long sequence in the batch delays all shorter ones.
• Idle Cycles: The GPU sits idle between batches if the queue is empty.

This makes it unsuitable for interactive applications like chatbots or real-time APIs, where tail latency (p95, p99) is the critical metric. It is the direct performance trade-off that continuous batching was invented to solve.

Understanding static batching is best done by contrasting it with its modern successor, continuous batching (or iteration-level scheduling).

Continuous batching effectively eliminates the major drawbacks of static batching for online serving.

Dynamic batching is a request scheduling strategy that groups multiple inference queries into a single batch based on a configurable batch window or queue size. It improves upon naive request-per-request processing but differs from continuous batching by typically forming a batch, processing it fully, and then forming a new one. It mitigates head-of-line blocking compared to static batching but can still incur wait times due to the fixed batch window. It is a common feature in inference servers to amortize the kernel launch overhead across multiple requests.

Padding is the process of adding dummy tokens to sequences within a batch so all sequences match the length of the longest one, enabling efficient parallel computation on GPUs. This creates significant memory and compute waste. Variable-length batching (or ragged batching) is a technique to minimize this overhead. It groups sequences of different lengths while using specialized kernels or data structures (e.g., NVIDIA's Ragged Tensor format) to process them without uniform padding. This is critical for optimizing the prefilling phase, where input prompt lengths vary widely.

Head-of-line blocking is a critical performance pathology in static and simple dynamic batching systems. It occurs when a single long-running or stalled request within a batch delays the processing and completion of all other requests in that same batch. This directly inflates tail latency (p95, p99) and degrades user experience. Continuous batching and advanced schedulers are designed specifically to mitigate this by allowing other sequences in the batch to complete and return independently.

An inference server (e.g., TensorFlow Serving, TorchServe, Triton) is the software system that hosts models and implements batching. A batching policy is the set of rules governing this process. For static batching, the policy is simple: wait for N requests or a timeout, then process. Advanced policies for dynamic/continuous batching consider:

• Maximum batch size (compute vs. memory limits)
• Batch timeout (latency vs. throughput trade-off)
• Request queue management
• Request admission control and load shedding to prevent system overload.

Understanding inference latency requires separating two distinct phases:

• Prefilling Phase: The initial, compute-bound parallel processing of the entire input prompt. Static batching can be highly efficient here if prompts are of similar length.
• Decoding Phase: The iterative, memory-bound generation of output tokens one-by-one. This is where static batching suffers most, as the entire batch remains locked until the longest sequence finishes generation. The iteration time for this phase dictates token generation latency. Optimizing the decoding phase is the primary driver for continuous batching techniques.