Batch Timeout

A batch timeout is a configurable duration that defines the maximum time an inference request can wait in the scheduler's queue before being dispatched for processing, even if the target batch size has not been reached. Its primary function is to bound tail latency by preventing requests from waiting indefinitely for a full batch to form, which is crucial for interactive applications like chatbots. This parameter creates a direct trade-off: shorter timeouts reduce latency but can lead to smaller, less efficient batches, while longer timeouts increase throughput at the cost of higher wait times.

The batch timeout parameter sits at the heart of the fundamental latency-throughput trade-off in inference serving.

• Low Timeout (< 50ms): Prioritizes latency. Requests are dispatched quickly, minimizing wait time in the queue. This is ideal for real-time applications but results in smaller batch sizes, lower GPU utilization, and reduced overall system throughput.
• High Timeout (> 200ms): Prioritizes throughput. The scheduler waits longer to accumulate more requests, forming larger batches that maximize GPU parallelization and compute efficiency. This increases the system's overall tokens/sec but introduces higher queuing delay, impacting p95 and p99 latency.

Batch timeout works in conjunction with other scheduler parameters to govern batch formation:

• Batch Window: Often used synonymously with batch timeout. It defines the temporal window for request accumulation.
• Maximum Batch Size: A hard limit on the number of requests in a batch, constrained by GPU memory (KV cache). The scheduler will dispatch a batch when either the timeout expires or the maximum batch size is reached, whichever comes first.
• Minimum Batch Size: Some systems may have a minimum size; if the timeout expires and fewer than the minimum requests are queued, they may still be held, or dispatched with potential inefficiency.

The setting directly influences hardware efficiency. An improperly configured timeout is a common source of idle cycles.

• Timeout Too Short: Results in frequent, small batches. The GPU is underutilized because the parallel compute units are not fully saturated. The fixed overhead of launching a computation kernel (kernel launch overhead) becomes a significant fraction of the total processing time.
• Timeout Too Long: While this maximizes batch size, it can leave the GPU idle while waiting for the timeout to expire if request arrival rate is low. The optimal timeout is dynamically tuned to match the request arrival pattern, keeping the GPU constantly busy with optimally sized batches.

Advanced inference servers implement adaptive timeouts that adjust based on real-time load to optimize the trade-off.

• Load-Based Adjustment: The timeout can be shortened during periods of high traffic (many requests/sec) as batches fill quickly anyway. It can be lengthened during low traffic to improve utilization.
• Request-Aware Scheduling: Timeouts can vary per request based on its Service Level Objective (SLO) or priority. High-priority user-facing requests may have a near-zero timeout, while background batch processing jobs tolerate a longer wait.
• Feedback Loops: Systems may use metrics like queue length and GPU utilization to automatically tune timeout values, moving towards autoscaling of scheduling parameters.

Batch timeout is a core feature of modern inference servers and is exposed as a key configuration parameter.

• vLLM: Configurable via the max_num_batched_tokens and scheduling policies, with timeout behavior implicit in its iteration-level scheduling.
• TensorRT-LLM: Batch scheduling parameters are set in the configuration, influencing how the scheduler groups requests over time.
• Triton Inference Server: Uses dynamic batching with a max_queue_delay_microseconds parameter, which is the batch timeout. The scheduler collects requests in the queue until the delay expires or the maximum batch size is met.
• Custom Schedulers: Often implement a main scheduling loop that checks the queue every few milliseconds, dispatching a batch if the oldest request has exceeded the timeout threshold.

The overarching scheduling strategy where an inference server groups multiple requests into a single batch based on a time window (batch window) or queue size. Unlike static batching, it allows for flexible batch formation to improve GPU utilization. Batch timeout is a critical parameter within this strategy, directly controlling the maximum wait time for request accumulation.

The configurable time duration an inference scheduler waits for incoming requests to accumulate before forming a new batch. It is the primary mechanism for implementing a batch timeout. A short window minimizes latency but may result in small, inefficient batches. A long window increases throughput potential but can degrade tail latency (p95, p99).

A buffer or holding area within the inference server where incoming client requests await scheduling. The batch timeout timer starts when a request enters this queue. Key queue management concepts include:

• Request Admission Control: Deciding whether to accept or reject a request based on system load.
• Backpressure: Signaling upstream clients to slow down request submission.
• Load Shedding: Deliberately dropping queued requests under extreme overload to protect system stability.

The fine-grained execution model used by continuous batching. Instead of a batch being fixed for an entire request, the scheduler can add new requests to the active batch and remove completed ones at each model iteration (token generation step). The batch timeout applies at the point a request seeks to join an executing batch, balancing the wait for a fuller batch against the urgency to start processing.

The high-percentile latencies of request completion (e.g., the 95th or 99th percentile). It is a critical user-facing metric for interactive applications. Batch timeout is a direct lever for controlling tail latency. An excessively long timeout can cause some requests to wait too long in the queue, inflating tail latency, even if average throughput improves.

The complete set of rules and heuristics governing how an inference scheduler forms and manages batches. The batch timeout parameter is a cornerstone of this policy. A comprehensive policy also considers:

• Maximum Batch Size: A hardware-driven limit to prevent out-of-memory errors.
• Priority Queues: Handling different service-level agreements (SLAs).
• Request Interleaving: Multiplexing requests with different sequence lengths to mitigate head-of-line blocking.