Continuous Batching

Unlike static batching, which waits for an entire batch to finish, continuous batching operates at the iteration level. As soon as a request within a batch completes its forward pass for a single token, that slot becomes available. A new waiting request is immediately inserted into this vacant slot for the next iteration. This creates a rolling batch where requests enter and exit the computational pipeline independently, dramatically improving GPU utilization and reducing tail latency for variable-length sequences.

Efficient management of the Key-Value (KV) cache is fundamental. Each request maintains its own cache of previously computed keys and values for the attention mechanism. Continuous batching requires a dynamic memory allocator to handle these caches as requests of different lengths join and leave the batch. Advanced systems like vLLM use PagedAttention, which stores the KV cache in non-contiguous, paged blocks. This eliminates memory fragmentation, allows efficient sharing of cached prompts, and supports much longer contexts—all essential for edge devices with constrained memory.

It's important to distinguish between two related techniques:

• Dynamic Batching: Groups multiple requests into a single batch before inference starts. All requests in the batch must be padded to the length of the longest sequence, leading to computational waste. Batches are formed on-the-fly but executed as a static unit.
• Continuous Batching: Eliminates padding waste by allowing requests to start and finish at different times within the same batch execution. It provides superior throughput and latency characteristics, especially for interactive, streaming applications common in edge RAG.

The primary technical benefit is near-optimal hardware utilization. By keeping the computational units (e.g., GPU SMX cores, NPU MAC units) constantly fed with work, it amortizes the fixed cost of loading model weights and maximizes FLOPs efficiency. This leads to significantly higher throughput (requests/second) compared to static batching. For edge deployments, this means serving more concurrent users or handling more complex RAG chains (retrieval + generation) on the same limited hardware, directly impacting total cost of ownership (TCO).

Continuous batching directly attacks latency, a critical metric for user-facing edge AI. It reduces:

• Queue Latency: Requests don't wait for a full batch to form.
• Compute Latency: No computational waste on padding tokens.
• Tail Latency (P99): Shorter requests aren't held hostage by longer ones in the same batch. For an edge RAG pipeline, this means faster end-to-end response times from the moment a user query is issued to the final generated answer, enhancing perceived performance and usability.

Dynamic batching is a precursor to continuous batching where an inference server groups multiple incoming requests into a single batch. Unlike static batching, it can handle requests of variable sequence lengths by padding them to the longest in the batch. However, the entire batch must finish processing before any results are returned, which can lead to head-of-line blocking and lower GPU utilization compared to continuous batching.

• Key Difference: Waits for a full batch vs. adding requests as others finish.
• Use Case: Suitable for more predictable, lower-variance workloads.

PagedAttention is a memory management algorithm for the key-value (KV) cache in transformer attention. It organizes the cache into non-contiguous, fixed-size blocks (pages), similar to virtual memory in operating systems. This drastically reduces memory fragmentation caused by variable-length sequences in continuous batches.

• Enables Continuous Batching: Efficient KV cache management is essential for supporting many concurrent, variable-length requests.
• Impact: Popularized by the vLLM inference engine, it allows for longer contexts and higher throughput in batched inference scenarios.

Model pipelining is a parallel execution strategy that splits a neural network across multiple hardware stages (e.g., different GPUs or NPU cores). In a RAG context, different stages could process the retriever, reranker, and generator components concurrently.

• Complementary to Batching: Works alongside continuous batching to improve overall system throughput.
• Edge Relevance: On heterogeneous edge chips, pipelining can keep different specialized units (CPU, NPU, GPU) busy simultaneously, hiding latency.

Compute offloading is a dynamic resource management strategy where parts of an inference pipeline are executed on different hardware tiers. For edge RAG, lightweight retrieval might run on-device, while the heavy LLM generation is offloaded to a nearby server or cloud.

• Relationship to Batching: Continuous batching optimizes the offloaded generator step on the server side.
• Goal: Balances low latency, privacy, and resource constraints by making intelligent where-to-compute decisions.

KV Cache Quantization reduces the precision (e.g., from FP16 to INT8 or INT4) of the Key-Value cache stored during autoregressive generation. This significantly decreases the memory footprint of each request in a batch.

• Directly Enables Larger Batches: By reducing per-request memory, more requests can be packed into a continuous batch, improving GPU utilization.
• Critical for Edge: Essential for running models with long context windows on memory-constrained edge hardware alongside continuous batching.