Query Batching

Batching transforms multiple sequential vector searches into a single, parallelized matrix operation. This saturates the massively parallel architecture of GPUs and AI accelerators (like TPUs), leading to significantly higher compute efficiency and throughput compared to processing queries one-by-one.

• Key Mechanism: Instead of computing query_vector • index_matrix for each query, the system computes batch_matrix • index_matrix.
• Impact: Can increase queries per second (QPS) by 10x or more on the same hardware, directly reducing infrastructure costs per query.

Every search query incurs fixed overhead costs: network latency, kernel launch latency on accelerators, and index loading into fast memory. Batching amortizes these fixed costs across many queries.

• Example: If kernel launch takes 50 microseconds, processing 100 queries sequentially incurs 5ms of pure overhead. Processing them in one batch incurs ~50 microseconds total.
• Benefit: This dramatically reduces the average latency per query in high-concurrency scenarios, making the system more scalable.

Modern hardware (GPUs, CPUs with AVX) is optimized for contiguous, aligned memory access patterns. Batched queries allow the system to load the vector index from memory (or cache) once for the entire batch, rather than making scattered, smaller accesses for each individual query.

• Technical Detail: This reduces memory bandwidth contention and improves cache hit rates.
• Result: Lower latency and more predictable performance, especially critical for meeting P99 latency service-level agreements (SLAs).

Effective batching requires intelligent scheduling. Systems implement a batch scheduler that collects incoming queries over a short time window (e.g., 5-50ms) before executing the batch.

• Trade-off Managed: The scheduler balances the latency penalty of waiting to form a batch against the throughput gain of a larger batch.
• Adaptive Systems: Advanced schedulers can adjust the batch window size dynamically based on current load to optimize total system throughput.

Query batching is particularly impactful in enterprise environments serving multiple users or applications simultaneously.

• Example Scenarios:
  • A customer support chatbot handling dozens of concurrent user sessions.
  • An internal search API serving multiple microservices.
  • A recommendation system generating predictions for a whole page of items at once.
• Outcome: Enables a single retrieval service node to handle a much higher number of concurrent requests without a linear increase in resource provisioning.

The primary technical driver for query batching is maximizing GPU utilization. Neural network inference on GPUs involves significant fixed overhead for kernel launches and memory transfers. By batching multiple queries, this overhead is amortized across all items in the batch.

• Parallel Matrix Operations: Embedding models and ANN search computations are expressed as large matrix multiplications. GPUs perform these operations much more efficiently on a single large batch matrix than on many small matrices sequentially.
• Kernel Launch Overhead: Each individual query requires launching GPU kernels. Batching reduces the frequency of these launches, minimizing latency from CPU-GPU synchronization.
• Memory Bandwidth Saturation: Transferring a single, contiguous batch of query vectors to GPU memory is more efficient than many small transfers, better utilizing the available memory bandwidth.

Batching strategies are categorized by how they handle incoming queries over time.

• Static Batching: Queries are collected for a fixed time window (e.g., 50ms) or until a fixed batch size (e.g., 32 queries) is reached, then processed. This is simple to implement but can introduce tail latency if the batch timer is the limiting factor.
• Dynamic Batching (Continuous Batching): A more advanced approach used in model serving frameworks like vLLM and Triton Inference Server. It allows new queries to join a batch that is already being processed for a model's prefill stage, and completed queries can exit the batch early during the decode stage. This maximizes GPU utilization with variable-length sequences and arrival times.
• Adaptive Batching: The batch size or window is adjusted dynamically based on system load, observed latency, and GPU utilization metrics to optimize the throughput-latency trade-off in real-time.

Query batching is implemented within serving frameworks and vector database clients.

• Client-Side Batching: The application code aggregates multiple user queries before sending a single request to the retrieval endpoint. This requires an asynchronous design pattern where queries can be deferred briefly.
• Server-Side Batching: The retrieval service (e.g., a vector database or a dedicated embedding service) automatically batches incoming concurrent requests. This is transparent to the client but requires stateful connection management.
• gRPC / HTTP/2 Streaming: Protocols that support multiplexing and streaming are used to efficiently send batched query vectors and receive batched results over the network, reducing connection overhead.
• Framework Examples: Milvus, Weaviate, and Qdrant support batched search requests via their client APIs. Model serving for embedding generation is commonly batched using TensorFlow Serving, TorchServe, or NVIDIA Triton.

Implementing batching introduces engineering trade-offs that must be carefully managed.

• Latency vs. Throughput: Batching improves aggregate throughput (queries per second) but can increase individual query latency for the first query that arrives and must wait for the batch to fill. The goal is to minimize this added latency while maximizing throughput gains.
• Batch Size Tuning: The optimal batch size is hardware-dependent (GPU VRAM capacity) and model-dependent (embedding dimension). It is found empirically by measuring throughput and P95/P99 latency under load.
• Heterogeneous Query Complexity: In a RAG system, not all queries trigger the same retrieval depth. Batching a simple lookup with a complex multi-hop query can lead to stragglers, where the entire batch waits for the slowest query. Techniques like query grouping by predicted complexity can mitigate this.
• Failure Semantics: If one query in a batch fails (e.g., due to a malformed input), the entire batch may fail. Robust implementations require error isolation and partial batch failure handling.

In a multi-stage retrieval pipeline, batching can be applied at each stage for compounded efficiency.

1. Embedding Generation: The first and most computationally intensive stage. Multiple raw text queries are batched through the embedding model (e.g., BAAI/bge-large-en).
2. ANN Search: The batched query vectors are simultaneously searched against the vector index. Libraries like Faiss and ScaNN have native support for batched search, performing parallel distance computations across all queries in the batch.
3. Reranking (Cross-Encoder): The candidate documents for all batched queries can be scored by a cross-encoder model in a single forward pass, which is highly efficient as these models are designed for sentence-pair classification.

This end-to-end batching minimizes data movement and kernel launches across the entire retrieval stack, providing the greatest overall latency reduction.

Effective batching requires monitoring specific performance indicators.

• Batch Utilization: The percentage of the maximum batch size that is typically used (e.g., avg. 28/32 slots). Low utilization suggests the batch window or size is too large for the current load.
• Batch Assembly Time: The time spent waiting for the batch to fill. This directly contributes to added latency.
• GPU Utilization: Track SM (Streaming Multiprocessor) activity and memory bandwidth. Effective batching should drive GPU utilization toward 70-100% during inference bursts.
• Throughput (QPS) vs. Latency Percentiles: The core trade-off curve. Plot P50, P95, and P99 latency against queries per second as batch size changes. The operational batch size is chosen based on the acceptable latency SLA.
• Tail Latency Impact: Monitor the difference between P50 and P99 latency. Poorly configured batching can cause this gap to widen significantly, leading to inconsistent user experience.

The use of Graphics Processing Units (GPUs) to parallelize the massive number of distance calculations and graph traversals in Approximate Nearest Neighbor (ANN) search. Libraries like FAISS and RAFT implement GPU kernels to achieve order-of-magnitude latency reductions for high-throughput retrieval.

• Parallelism: Exploits thousands of GPU cores to compute distances between a query and millions of candidate vectors simultaneously.
• Use Case: Essential for real-time retrieval over billion-scale vector indexes where CPU-based search is prohibitive.

A programming model where the search operation is non-blocking, allowing a system's main thread or other workers to handle additional requests while waiting for I/O-bound operations (e.g., disk access, network calls to a vector database). This improves overall system throughput and resource utilization.

• Contrast with Synchronous: In synchronous retrieval, the thread is blocked until the search completes, limiting concurrency.
• Implementation: Often uses async/await patterns or event loops, common in frameworks like Python's asyncio.

An in-memory store (e.g., Redis, Memcached) that holds pre-computed vector embeddings for frequently accessed documents or user queries. It eliminates the need to recompute embeddings via a model inference call, a significant source of pre-retrieval latency.

• Cache Key: Typically a hash of the document text or query string.
• Performance Impact: Can reduce retrieval latency by hundreds of milliseconds, especially for repetitive queries in enterprise settings.

A cascaded architecture that optimizes the precision-latency trade-off by using a fast, approximate first-stage retriever (e.g., BM25 or ANN) to fetch a broad candidate set, which is then re-ranked by a slower, more accurate model (e.g., a cross-encoder).

• Latency Benefit: The expensive re-ranker only processes 100-1000 candidates, not the entire corpus.
• System Design: Query batching is often applied independently to each stage based on its computational profile.