Slow Query Log

The execution time threshold is the configurable duration that defines a 'slow' query. When a query's runtime exceeds this threshold, it is captured in the log. This setting is crucial for filtering operational noise from genuine performance issues.

• Dynamic Adjustment: Thresholds can be set globally or per-index based on expected performance Service Level Objectives (SLOs).
• Example: Setting a threshold of 100ms for a production semantic search API ensures only queries degrading user experience are logged, ignoring fast-enough searches.

The log captures the complete query context, which is essential for reproducing and diagnosing issues. This includes:

• The Query Vector: The embedding used for the similarity search.
• Search Parameters: The exact k (number of nearest neighbors), distance metric (e.g., cosine, L2), and any filter predicates applied.
• Client Metadata: Source IP, user ID, or application name to trace the query origin.
• Timestamp: Precise time of query execution for correlation with other system events.

Entries detail the specific vector index accessed and system resource consumption during the query's execution. This helps identify bottlenecks related to specific data segments or hardware limits.

• Index Segment ID: Identifies which part of a partitioned or sharded index was queried.
• Resource Metrics: CPU time, memory allocated, and I/O wait time for disk-based indices.
• Cache Performance: Notes whether the query was a cache hit or cache miss, explaining cold start latency.

For advanced vector databases, the log may include a query plan explanation. This describes the algorithmic path taken to execute the search, which is vital for optimization.

• Algorithm Used: Indicates if the search used HNSW, IVF, or a brute-force sequential scan.
• Traversal Details: For graph-based indices like HNSW, it may log the number of nodes visited or graph layers traversed.
• Filter Evaluation Order: Shows how metadata filters were applied—before, during, or after the vector search (pre-filter, post-filter, or single-stage).

Beyond timing, the log can capture diagnostics about the results returned, linking performance to output quality.

• Actual Recall: The proportion of true nearest neighbors found versus expected, if ground truth is available for validation.
• Result Cardinality: The number of results returned after applying filters, which can indicate overly restrictive queries.
• Distance Scores: The similarity scores of returned vectors, helping identify if the query is searching in a sparse or dense region of the vector space.

Slow query logs are not isolated files; they feed into broader observability systems. This enables trend analysis and alerting.

• Export Formats: Logs are typically written in structured formats like JSON for easy ingestion by tools like Datadog, Grafana Loki, or Elasticsearch.
• Metric Derivation: Log data is aggregated to create dashboards tracking p95/p99 query latency and slow query rate over time.
• Alerting: Can trigger alerts when the rate of slow queries spikes, indicating a potential system degradation or configuration drift.

The automated collection, transmission, and measurement of operational data from a vector database system. This encompasses the three pillars of observability: metrics (e.g., QPS, latency), logs (including the slow query log), and distributed traces. Telemetry data is essential for creating dashboards, setting alerts, and performing root cause analysis on performance degradations.

A formal, quantitative target for the accuracy of a vector database's similarity search. It defines the minimum acceptable proportion of true nearest neighbors that must be successfully returned over a measurement period (e.g., "99.9% recall over 30 days"). The Slow Query Log is a primary tool for investigating breaches of latency SLOs, while recall SLOs are validated through offline benchmarking and A/B testing of index parameters.

A defensive stability mechanism where a vector database under excessive load intentionally rejects or delays lower-priority incoming queries. This prevents a cascading failure and protects core functionality for high-priority requests. The Slow Query Log helps identify the query patterns and resource consumption that trigger load shedding, informing capacity planning and query optimization to avoid the condition.

A key performance metric measuring the percentage of similarity search requests served from an in-memory cache versus requiring a disk read. A low hit ratio directly contributes to queries appearing in the Slow Query Log. Optimizing this ratio involves tuning cache size, eviction policies (e.g., LRU), and data access patterns.

The elevated query response time experienced when a vector index segment is first loaded from disk into memory. Queries during this phase are prime candidates for the Slow Query Log. Mitigation strategies include:

• Pre-warming: Loading indexes during startup or maintenance windows.
• Pinning: Keeping critical index segments permanently in memory.
• Progressive loading: Staggering the load of large indexes.

A stability pattern that temporarily stops calling a failing downstream service (e.g., an external embedding model API) after a failure threshold is reached. While not a direct log, a Slow Query Log may show queries timing out due to a tripped circuit breaker. This pattern prevents system resource exhaustion and allows the failing service time to recover, turning a flood of slow failures into a clean, fast-failing state.