Circuit Breaker

The core logic of a circuit breaker is implemented as a finite state machine with three distinct states:

• Closed: Requests flow normally to the service. Failures are counted.
• Open: The circuit trips after a failure threshold is exceeded. All requests fail fast without calling the service.
• Half-Open: After a timeout, a limited number of test requests are allowed to probe if the service has recovered. Success resets the circuit to Closed; failure returns it to Open.

The breaker monitors for specific failure conditions to decide when to trip. Key configurable thresholds include:

• Failure Count: The number of consecutive failures (e.g., timeouts, 5xx errors) required to open the circuit.
• Failure Ratio: The percentage of failed requests within a sliding time window.
• Timeout Duration: Individual request timeouts that count as failures. For vector databases, this is crucial when calling external embedding APIs which may hang.

When the circuit is Open, calls fail immediately without network latency. This fail-fast behavior reduces load on the failing service and the calling system. Implementations should provide a fallback mechanism, such as:

• Returning a cached or default response (e.g., a generic embedding).
• Queuing the request for later retry.
• Failing gracefully to the user with a meaningful error. This prevents thread pool exhaustion in the calling application.

The Half-Open state enables automatic recovery. After a configured resetTimeout, the circuit allows one or a few test requests through:

• If successful, the circuit assumes the service is healthy and resets to Closed.
• If the test fails, the circuit returns to Open for another full timeout period. This probe mechanism prevents the system from flooding a recovering service with traffic the moment it comes back online.

A production-grade circuit breaker emits detailed telemetry for observability:

• State Transition Metrics: Logs when the circuit opens, closes, or goes half-open.
• Request Counts: Tracks successful, failed, and short-circuited (failed-fast) requests.
• Latency Histograms: Measures call durations. This data is essential for SLO/SLI calculation and understanding the health of dependent services like embedding models or external vector APIs.

In a clustered vector database or microservices architecture, a local circuit breaker state may be insufficient. Distributed coordination ensures all nodes share a consistent view of a downstream service's health. This can be achieved via:

• Gossip protocols to propagate state.
• Centralized state in a coordination service like Redis or etcd.
• Without coordination, partial failures can lead to inconsistent client experiences and reduced effectiveness of the pattern.

A primary use case is guarding the embedding model API that generates vectors for a database. If the model endpoint times out or returns errors (e.g., 5xx HTTP status), the circuit breaker trips. This prevents the vector database ingestion pipeline from being blocked by a downstream failure, allowing it to queue requests or use a fallback model. It directly protects the vector indexing process from stalling.

In a distributed vector database cluster, a circuit breaker can be applied to individual nodes. If a replica node becomes slow or unresponsive due to high memory pressure or disk I/O issues, the client-side or load balancer circuit breaker stops routing queries to it. This enables failover to healthy nodes, maintains overall query latency SLOs, and gives the faulty node time to recover or be replaced without bringing down the entire service.

In a Retrieval-Augmented Generation (RAG) system, a circuit breaker protects the interaction between the retrieval step (vector search) and the generation step (LLM). If the vector database query latency spikes beyond a threshold—indicating potential index corruption or overload—the circuit breaker can fail fast. This allows the system to return cached results, degrade gracefully to keyword search, or return a user-friendly message instead of timing out the entire user request.

For hybrid search systems that combine vector similarity with metadata from an external knowledge graph, circuit breakers are essential. If the graph database query fails or is too slow, the breaker trips after a configured number of failures. This ensures the core vector similarity search remains functional, even if the enriched contextual filtering is temporarily unavailable, preserving system availability.

During large-scale batch ingestion of vectors, a circuit breaker monitors the health of the destination database. If write errors or backpressure exceed a limit (e.g., due to hitting storage quotas or rate limits), the breaker opens. This pauses the ingestion job, preventing a flood of retries that could exacerbate the problem. It allows operators to intervene and scale resources before resuming, aligning with data management and recovery point objectives (RPO).

A circuit breaker in the vector database API layer protects upstream AI agents or applications. If the database is overwhelmed (e.g., from a slow query storm), the breaker trips and quickly rejects new requests instead of letting them queue. This load shedding prevents thread exhaustion in the calling services, stopping a localized database issue from cascading into a widespread application failure. It is a key pattern for agentic observability and system resilience.

A defensive mechanism where a system under excessive load intentionally rejects or delays non-critical incoming requests to prevent a total failure. Unlike a circuit breaker, which targets a specific failing dependency, load shedding protects the system's own resources.

• Purpose: Prioritizes core functionality and prevents cascading failure from resource exhaustion.
• Mechanism: Uses admission control policies (e.g., rate limiting, queue management) to drop excess traffic.
• Example: A vector database might reject new query connections during a traffic spike to protect the integrity of ongoing indexing jobs.

A fault-handling pattern where a failed operation is automatically retried after a delay, with the delay increasing exponentially between attempts. This is often used in conjunction with a circuit breaker.

• Purpose: To handle transient faults (e.g., network timeouts) by giving a failing service time to recover.
• Key Parameters: Maximum retry count, initial backoff delay, and backoff multiplier.
• Interaction with Circuit Breaker: Retries should stop when the circuit is OPEN to avoid hammering the failed service.

A resilience pattern that isolates elements of an application into pools, so a failure in one pool does not drain resources and cause a system-wide failure. It's analogous to watertight compartments in a ship.

• Purpose: To limit the blast radius of a failure and preserve partial service availability.
• Implementation: Uses separate thread pools, connection pools, or even processes for different client classes or operations.
• Vector DB Example: Isolating query traffic from backup traffic, so a slow backup doesn't block all user searches.

A dedicated API endpoint (e.g., /health) that returns the operational status of a service. It is the primary signal used by upstream systems, including circuit breakers and orchestrators, to assess liveness.

• Function: Returns HTTP status codes (200 for healthy, 503 for unhealthy) and optionally detailed component status.
• Usage: Circuit breakers may call this endpoint during the HALF-OPEN state to test if a service has recovered.
• Orchestration: Used by Kubernetes for liveness and readiness probes to manage container lifecycles.

A control mechanism that restricts the number of requests a client can make to a service within a given time window. It protects downstream services from being overwhelmed, complementing a circuit breaker's role.

• Objective: Prevent resource starvation, ensure fair usage, and manage quotas.
• Common Algorithms: Token bucket, fixed window, sliding window log.
• Distinction: Rate limiting is proactive and policy-based, while a circuit breaker is reactive and failure-based.

A fundamental reliability mechanism that sets a maximum duration for an operation to complete. If the deadline is exceeded, the operation is abandoned, freeing up resources.

• Purpose: To prevent calls from waiting indefinitely for a non-responsive service.
• Architectural Layer: Should be applied at every network boundary (client, service, database driver).
• Relationship to Circuit Breaker: Repeated timeouts are a primary failure signal that can trip a circuit breaker into the OPEN state.