Circuit Breaker

The core logic of a circuit breaker is a finite state machine with three distinct states:

• Closed: Requests flow normally to the service. Failures are counted.
• Open: The circuit 'trips.' All requests fail immediately without reaching the service.
• Half-Open: After a timeout, a limited number of test requests are allowed to probe if the service has recovered. Success transitions back to Closed; failure returns to Open.

The breaker monitors requests for failures (timeouts, 5xx HTTP errors, connection errors). It trips from Closed to Open when a configured threshold is exceeded within a sliding time window. Common configurations include:

• Failure Count Threshold: Trip after N consecutive failures.
• Failure Ratio Threshold: Trip when X% of the last Y requests fail.
• Slow Call Threshold: Trip when requests exceed a latency threshold, treating them as failures.

After the circuit has been Open for a configured reset timeout, it enters the Half-Open state. This is a probationary period where:

• A single request or a small batch is allowed to pass through.
• Its success or failure determines the next state.
• Success closes the circuit, resuming normal operation.
• Failure re-opens the circuit, restarting the timeout. This mechanism allows the system to automatically detect when a backend service (e.g., a vector database API) has recovered without requiring manual intervention.

When the circuit is Open, requests fail immediately at the client SDK level. This fail-fast behavior:

• Reduces load on the failing service.
• Prevents thread/connection pool exhaustion in the calling service.
• Provides predictable, low-latency failure responses.

SDKs often pair this with fallback logic, such as returning cached results, default values, or queuing requests for later retry, maintaining partial functionality for the user.

Circuit breakers and retry logic are complementary but distinct patterns. They are often layered:

1. Retry handles transient, sporadic failures (e.g., network blip) for individual requests.
2. Circuit Breaker handles persistent, systemic failures affecting many requests.

A robust SDK will retry a request with backoff while the circuit is Closed. If failures persist and trip the circuit, all subsequent requests fail-fast until recovery is probed. Misconfiguration (e.g., aggressive retries without a breaker) can exacerbate downstream failures.

Effective circuit breakers expose metrics for system observability, crucial for SREs and DevOps engineers. Key metrics include:

• Circuit State (Closed/Open/Half-Open)
• Failure Count and Failure Rate
• Request Count (total, successful, failed)
• State Transition Events (e.g., CLOSED → OPEN)

These metrics are typically exported to monitoring systems like Prometheus, allowing teams to visualize system health, set alerts for prolonged outages, and tune breaker thresholds (e.g., timeout duration, failure ratio) based on empirical data.

A circuit breaker operates through three distinct states, transitioning based on failure thresholds and timeouts.

• CLOSED: The normal state. Requests flow to the vector database API. Failures are counted. If failures exceed a configured threshold within a time window, the breaker trips to OPEN.
• OPEN: All requests fail immediately without reaching the API. A timer is set. After this reset timeout elapses, the breaker moves to HALF-OPEN.
• HALF-OPEN: A limited number of trial requests are allowed to pass. Their success or failure determines the next state: success moves back to CLOSED; failure returns to OPEN.

The breaker must accurately detect failures to trip appropriately. Common configurations include:

• Failure Count Threshold: Trip after N consecutive failures (e.g., 5 failed query calls).
• Failure Ratio Threshold: Trip if X% of the last Y requests fail (e.g., 50% of last 20 requests).
• Timeout Detection: Treat any request exceeding a latency threshold (e.g., 2 seconds) as a failure, even if it would eventually succeed.

These thresholds are applied to specific error types. For vector databases, network timeouts, 5xx server errors, and connection refusals typically count as failures, while 4xx client errors (like invalid query syntax) often do not.

Circuit breakers and retry logic are complementary but distinct patterns. They must be layered correctly to avoid conflict.

• Retry Logic handles transient failures (e.g., a momentary network blip) by reattempting the same request.
• Circuit Breaker handles persistent failures (e.g., the database is down) by blocking all requests.

Best Practice: Implement retry logic inside the closed circuit. If retries exhaust and fail, they contribute to the breaker's failure count. Once the breaker is OPEN, no retries should be attempted for new requests, as they would guaranteed to fail.

When the circuit is OPEN, the SDK should provide a fallback response instead of just throwing an error. Common strategies include:

• Static Fallback: Return a default, empty, or cached result (e.g., an empty list for a search query).
• Degraded Service: Route the request to a backup or read-replica cluster if available.
• Queue for Later: Buffer write operations (upsert, delete) in a local queue to be replayed when the circuit closes.

Providing a fallback maintains application functionality, even if in a degraded mode, and is crucial for user experience.

Circuit breaker state changes are critical operational events that must be monitored.

Key metrics to expose and alert on:

• Breaker state transitions (CLOSED → OPEN, OPEN → HALF-OPEN).
• Current failure count/ratio.
• Number of requests rejected while OPEN.
• Latency of trial requests in HALF-OPEN state.

These metrics should be integrated into the SDK's telemetry, allowing Site Reliability Engineers (SREs) to correlate API outages with breaker activity and tune thresholds (like resetTimeout) based on observed recovery times of the vector database.

A conceptual example of circuit breaker integration in a vector database client SDK.

pythonCopy
class VectorDBClient:
    def __init__(self):
        self.circuit_breaker = CircuitBreaker(
            failure_threshold=5,
            reset_timeout=30, # seconds
            expected_exceptions=(ConnectionError, TimeoutError, ServerError)
        )

    @circuit_breaker
    def query(self, vector, top_k=10):
        # This call is wrapped/protected by the breaker
        response = self._http_post("/query", json={"vector": vector, "top_k": top_k})
        return response.json()

    def query_with_fallback(self, vector, top_k=10):
        try:
            return self.query(vector, top_k)
        except CircuitBreakerOpenError:
            # Fallback when the breaker is OPEN
            logging.warning("Circuit OPEN. Returning cached results.")
            return self._get_cached_results(vector, top_k)

This pattern decouples failure detection from business logic, making the client resilient.

A predefined alternative action or response when a primary operation fails and a circuit breaker is open. It maintains some level of functionality during outages.

• Static Response: Returning cached data or a default value.
• Degraded Functionality: Switching to a less accurate but available backup service (e.g., a keyword search fallback for a vector search).
• Graceful Degradation: Designing systems to fail softly, providing partial results instead of complete failure.