Load Shedding

Load shedding is a proactive mechanism. Instead of waiting for resources to be completely exhausted—leading to cascading failures, high latency, or crashes—the system preemptively rejects traffic. This contrasts with reactive failures like out-of-memory errors or timeouts, which are harder to recover from. The goal is to maintain a graceful degradation of service, protecting core functionality like existing query completion and data durability.

The decision to shed load is governed by admission controllers that monitor system health. Key configurable thresholds include:

• CPU Utilization: Queries are rejected when CPU usage exceeds a set percentage (e.g., 90%).
• Memory Pressure: Shedding triggers when available RAM for the vector index or query processing falls below a threshold.
• Queue Depth: Limits the number of pending queries in internal buffers.
• Concurrent Connections: Caps the number of active client connections. These parameters allow Site Reliability Engineers (SREs) to tune the system's behavior based on observed capacity and Service Level Objectives (SLOs).

Different strategies determine which requests are shed and how clients are notified:

• Random Drop: Simple but unfair; randomly rejects incoming requests.
• Oldest First: Drops requests that have been queued the longest.
• Priority-Based: Uses client-supplied or query-type priorities (e.g., read-only vs. write queries). Writes are often protected over reads to ensure data durability.
• Latency-Based: Rejects queries predicted to exceed a latency SLO. The system typically responds with an HTTP 503 Service Unavailable status code or a gRPC UNAVAILABLE error, signaling the client to retry with backoff.

Load shedding works in concert with broader infrastructure orchestration patterns:

• Circuit Breakers: Client-side circuit breakers detect 503 responses and stop sending traffic to the overloaded node, allowing it to recover. This creates a cooperative feedback loop.
• Health Checks: Load balancers and orchestrators like Kubernetes use liveness and readiness probes. A node under extreme load may fail its readiness probe, causing the orchestrator to temporarily remove it from the service pool, effectively shedding load at the routing layer.
• Autoscaling: Persistent load shedding can trigger autoscaling policies to add more compute nodes to the vector database cluster.

In multi-tenant vector database deployments, where a single cluster serves multiple independent clients or applications, load shedding is essential for noisy neighbor isolation. A surge in queries from one tenant could otherwise degrade performance for all others. Admission control can be applied per-tenant to enforce resource quotas. This ensures one tenant's overload does not breach the SLOs of others, a key requirement for SaaS vector database offerings.

Effective load shedding requires detailed telemetry. Key metrics to monitor include:

• Shedded Requests Per Second: Volume of rejected traffic.
• Admission Controller Status: Current state of admission gates (open/closed).
• System Resource Utilization: CPU, memory, and I/O at the time of shedding events.
• Client Retry Rates: An increase can indicate shedding is occurring. These metrics should be integrated into dashboards and alerts. A well-tuned system will show a sharp increase in shedded requests as utilization hits its threshold, acting as a pressure relief valve visible to operators.