Throughput (QPS)

Throughput and latency exhibit a fundamental trade-off. As the offered load (queries per second) increases towards a system's maximum throughput, queuing delays accumulate, causing average and tail latency (e.g., P99 Latency) to rise non-linearly. The goal is to operate at a throughput level that keeps latency within Service Level Objectives (SLOs). Pushing beyond the saturation point leads to thrashing and exponential latency growth.

High throughput is achieved by processing multiple queries simultaneously. This relies on:

• Concurrency: The system's ability to handle multiple in-flight queries through techniques like asynchronous I/O and non-blocking operations.
• Parallelism: The physical execution of multiple queries at the same instant, utilizing multi-core CPUs and SIMD (Single Instruction, Multiple Data) instructions for batched distance computations. Effective load balancing across threads and nodes is critical to prevent bottlenecks and maximize parallel resource utilization.

Maximum sustainable throughput is determined by the first resource to reach 100% utilization, known as the bottleneck. Common saturation points in vector databases include:

• CPU Cores: Exhausted by distance calculations and graph traversal logic.
• Memory Bandwidth: Overwhelmed by streaming large volumes of vector data from RAM.
• Network I/O: Saturated in distributed clusters during query fan-out and result aggregation.
• Disk I/O: Becomes a bottleneck for persistent indexes that are not fully memory-resident. Profiling identifies the limiting resource for targeted optimization.

The choice of Approximate Nearest Neighbor (ANN) index and its parameters directly dictates the throughput-latency frontier. For example:

• HNSW: Higher M (maximum connections) and efConstruction parameters create a more accurate, denser graph, increasing per-query latency but potentially allowing higher throughput at a given recall target due to more efficient search paths.
• IVF indexes: A larger nlist (number of Voronoi cells) reduces the number of vectors per cell, speeding up individual queries (lower latency) but requiring more distance computations to centroids, which can impact throughput if not optimized via Inverted File Index pruning.

Processing multiple queries in a single batch is a primary technique for maximizing throughput. It amortizes system overheads and enables hardware-efficient operations:

• Batch Size: The number of queries grouped together. Larger batches increase throughput but also increase the latency for the first query in the batch.
• Hardware Utilization: Enables full utilization of SIMD units on CPUs and tensor cores on GPUs by performing many distance calculations in parallel.
• System Overhead: Reduces per-query costs for network serialization, scheduling, and index access. Continuous batching, where batches are dynamically assembled from a queue, is used in high-throughput serving systems.

Throughput is not a single number but a curve dependent on load and latency constraints. It is characterized through:

• Load Testing: Systematically increasing the request rate until latency SLOs are violated or resource saturation occurs.
• Percentile Analysis: Throughput must be evaluated alongside P50, P95, and P99 Latency metrics. A system may sustain 10k QPS at P50 latency <10ms but only 5k QPS at P99 latency <50ms.
• Steady-State vs. Burst: Sustainable throughput is the rate a system can maintain indefinitely, distinct from short-term burst capacity which may rely on transient resource buffers.

Query Latency is the elapsed time between submitting a search request to a vector database and receiving the complete response. It is the primary user-facing metric for responsiveness.

• Direct Trade-off with Throughput: Systems are typically tuned for either low latency (prioritizing individual request speed) or high throughput (prioritizing total request volume). Increasing load often increases latency.
• Measurement Points: Includes network transmission, query planning, index traversal, distance computations, and result serialization.
• Service-Level Objectives (SLOs): Defined using percentiles (e.g., P95, P99 Latency) to guarantee performance for most queries.

P99 Latency is the 99th percentile of query latency measurements, representing the worst-case latency experienced by all but the slowest 1% of queries. It is crucial for defining reliable SLOs.

• Tail Latency Focus: Highlights outliers caused by garbage collection, cache misses, OS scheduling, or "herd" effects during concurrent load.
• Throughput Relationship: As QPS approaches system limits, P99 latency typically increases non-linearly, signaling saturation.
• Engineering Priority: Reducing P99 often requires deeper system optimization than improving average or median latency.

Recall is the fraction of true nearest neighbors (from an exhaustive search) that are successfully retrieved by an approximate search algorithm. It measures the completeness of search results.

• Accuracy/Performance Trade-off: Higher recall typically requires searching more of the index, increasing latency and reducing potential QPS. Tuning involves lowering recall targets to gain throughput.
• Recall@K: The standard metric, defined as the proportion of the true top-K neighbors found in the retrieved top-K results.
• Business Impact: Low recall means missed relevant items, which can degrade application quality (e.g., poor recommendations).

System Saturation occurs when a vector database can no longer increase QPS without violating latency SLOs or when latency increases exponentially with load. Load Testing is the process of characterizing this limit.

• Finding the Knee: The goal is to identify the maximum sustainable QPS before latency becomes unacceptable.
• Tools & Methods: Uses tools like wrk2 or ghz to generate controlled load while measuring latency distributions and error rates.
• Capacity Planning: Establishes the relationship between QPS, latency, recall, and hardware resources (CPU, memory, I/O).

Concurrency refers to the number of simultaneous queries being processed. Connection Pooling is a client-side mechanism to manage reusable network connections to the database, critical for achieving high QPS.

• Overhead Management: Establishing a new TCP/TLS connection per query is prohibitively expensive. Pools amortize this cost.
• Pool Sizing: The optimal pool size is related to the database's ability to handle concurrent requests (its thread/goroutine pool). Too small a pool limits QPS; too large can overwhelm the server.
• Throughput Enabler: Efficient concurrency management is a prerequisite for maximizing QPS.

Batching is the technique of combining multiple independent queries into a single request to the vector database. It is a powerful method to increase effective throughput and reduce per-query overhead.

• Throughput Amplification: A single network round-trip and query planning overhead can service N queries, dramatically increasing QPS capacity.
• Server Optimization: Databases can use continuous batching internally to group independent queries for parallelized index traversal and distance computations.
• Use Case: Essential for offline processing, embedding generation, and batch inference pipelines where latency per individual query is less critical than total job completion time.