Batch Size

Batch size directly mediates the classic trade-off between throughput (samples processed per second) and latency (time to process a single request).

• Larger batches maximize hardware parallelism, saturating GPU/TPU compute units to achieve higher throughput. This is ideal for offline processing.
• Smaller batches reduce the time-to-first-token for individual requests, minimizing latency for interactive applications like chatbots.

In inference, techniques like continuous batching aim to optimize this trade-off dynamically.

Batch size is a primary determinant of peak memory usage during both training and inference. Memory scales approximately linearly with batch size for:

• Activations: Intermediate values stored during the forward pass for backpropagation (training).
• Key-Value (KV) Cache: In transformer inference, the cache storing attention states for generated tokens. Larger batches require more cache memory.
• Gradients (training only): Accumulated gradients for each parameter.

Exceeding GPU memory (VRAM) limits causes out-of-memory (OOM) errors, making batch size a critical constraint.

In training, batch size influences optimization dynamics and final model quality.

• Small Batches: Provide a noisy gradient estimate, which can act as a regularizer and help models converge to flatter, potentially more generalizable minima. However, they require more iterations to converge.
• Large Batches: Provide a more accurate gradient estimate, enabling faster convergence per epoch but risk converging to sharp minima with poorer generalization. This may require adjusting other hyperparameters like learning rate.

The relationship is summarized by the generalization gap, where very large batches can sometimes lead to reduced test accuracy.

Batch size determines how effectively modern parallel hardware is utilized.

• Data Parallelism: The most common form, where the batch is split across multiple devices (e.g., GPUs). Larger batches improve compute utilization by keeping thousands of arithmetic logic units (ALUs) busy, reducing idle cycles.
• Underutilization: Small batches fail to fully utilize the massive parallelism of accelerators, leaving hardware memory-bound or idle, drastically reducing FLOPS/TOPS efficiency.

Optimal batch size is often the largest that fits in memory without causing other degradations.

Batch size can affect the numerical behavior of training, particularly for normalization layers.

• Batch Normalization: Relies on per-batch statistics (mean/variance). With very small batches (e.g., size 1 or 2), these statistics become noisy and unreliable, degrading performance. Large batches provide stable statistics.
• Alternative Normalizations: For small-batch scenarios, techniques like Layer Normalization, Group Normalization, or Instance Normalization are used, as they do not depend on batch dimensions.
• Gradient Precision: In mixed-precision training, larger batches can improve the numerical stability of gradient accumulation in FP16/BF16 formats.

During model serving, batch size is a key lever for inference cost optimization.

• Static vs. Dynamic Batching: Static batching uses a fixed size, causing delays. Dynamic batching groups requests from a queue, optimizing size in real-time.
• Continuous Batching (Iteration-Level Scheduling): Advanced technique where sequences in a batch are processed independently; new requests can join and finished ones can exit at each decoding step, maximizing GPU utilization despite variable request lengths.
• Tail Latency Impact: Poor batching policies can cause head-of-line blocking, where a single long sequence delays an entire batch, spiking p99 latency.

An inference optimization technique that dynamically groups incoming requests into batches at each model iteration. Unlike static batching, it allows new requests to join and completed sequences to exit the batch without waiting for the entire original batch to finish. This maximizes GPU utilization and improves throughput for interactive workloads with variable request arrival times and sequence lengths.

• Core Mechanism: The scheduler maintains a pool of active requests and re-evaluates the batch composition before every model forward pass.
• Contrast with Static Batching: Eliminates the idle time where a GPU waits for a fixed batch to complete before starting a new one.

A request scheduling strategy that groups multiple inference queries into a single batch based on a time window or queue size threshold. It is a broader category that includes continuous batching. The scheduler waits for a configured batch_window (e.g., 10ms) or until a queue reaches a certain size before forming and dispatching a batch.

• Key Parameter: The batch_window directly trades off latency (shorter window) for throughput (longer window).
• Implementation: Found in inference servers like TensorFlow Serving and Triton Inference Server.

The fine-grained scheduling approach used by continuous batching, where decisions are made per decoding step. The scheduler determines which requests are active, which have finished generation, and how to organize the batch's tensor inputs for the next forward pass. This requires sophisticated memory management for the KV Cache.

• Granularity: Operates at the level of a single token generation step.
• Challenge: Must efficiently handle ragged tensors (sequences of different lengths) to minimize padding and memory waste.

Techniques to handle sequences of different lengths within a fixed-size batch tensor. Padding adds dummy tokens to shorter sequences so all have the same length, enabling parallel computation but wasting FLOPs on the padded tokens. Variable-length batching (or ragged batching) uses specialized kernels or data structures to process sequences without uniform padding, improving efficiency.

• Padding Overhead: Can lead to significant computational waste, especially with highly variable sequence lengths.
• Optimization: Advanced inference engines use kernel fusion and custom CUDA kernels to execute variable-length batches efficiently.

The two distinct computational phases in autoregressive model inference that are affected differently by batch size.

• Prefill (Prompt Processing): The initial, compute-bound phase where the entire input prompt is processed in parallel. Larger batch sizes here fully utilize GPU compute cores but require substantial memory for the initial KV Cache.
• Decoding (Token Generation): The iterative, memory-bandwidth-bound phase where tokens are generated one per step. Batch size here determines the number of concurrent sequences sharing memory bandwidth. Increasing it improves throughput but can increase per-request latency due to iteration time.

The fundamental engineering trade-off governed by batch size. Throughput (tokens/second) generally increases with larger batch sizes due to better hardware utilization and amortization of fixed costs. Latency (time-to-first-token, time-per-output-token) often increases with batch size due to longer queue times and increased iteration_time during decoding.

• Optimization Goal: The choice of batch size (static or dynamic) is a direct lever to prioritize one metric over the other.
• Tail Latency: Large batches can cause high p95/p99 latencies due to head-of-line blocking, where one slow request delays an entire batch.