Circuit Breaker

A circuit breaker operates as a deterministic state machine with three distinct states:

• Closed: Requests flow normally to the downstream service. A failure counter tracks consecutive errors.
• Open: When the failure threshold is breached, the breaker trips. All subsequent requests are immediately rejected with a fallback response, giving the downstream service time to recover.
• Half-Open: After a configurable reset timeout, a limited number of probe requests are allowed through. If they succeed, the breaker resets to Closed; if they fail, it returns to Open.

This prevents the thundering herd problem where a recovering service is immediately overwhelmed by queued requests.

The sensitivity of a circuit breaker is governed by two primary parameters that must be tuned to the specific latency profile of the retrieval pipeline:

• Error Rate Threshold: The percentage of failed requests (e.g., 50%) over a sliding time window that triggers the Open state. This must be set above normal transient error rates.
• Slow Call Threshold: A secondary trigger based on P99 latency exceeding a defined Service Level Objective (SLO). If the downstream ANN index responds too slowly, the breaker can trip even without explicit errors.

Misconfiguration leads to either false positives (unnecessary tripping) or false negatives (failing to protect the system).

When a circuit breaker is Open, it must return a deterministic fallback to avoid propagating errors upstream to the answer generation engine:

• Stale Cache: Serve the last known good result from a semantic cache or KV-cache, accepting reduced freshness for availability.
• Degraded Retrieval: Fall back to a simpler, faster retrieval method, such as switching from a dense vector search to a sparse BM25 keyword search.
• Graceful Degradation: Return an empty result set with a clear error signal, allowing the orchestrator to synthesize an answer from remaining available sources.

The fallback must never throw an unhandled exception.

Without a circuit breaker, a slow downstream embedding service causes cascading thread exhaustion. Upstream callers retry aggressively, creating a retry storm that amplifies load on the already-failing service.

The circuit breaker acts as a backpressure mechanism. By failing fast in the Open state, it sheds load immediately, consuming minimal resources (a simple state check) rather than holding threads open waiting for timeouts.

This is often combined with jitter on retry intervals in the Half-Open state to de-synchronize probe requests from multiple service instances.

Circuit breakers are placed at critical choke points in the Answer Engine Architecture:

• Embedding Service: Protects against latency spikes when converting user queries to vectors.
• Vector Database (ANN Index): Trips when HNSW graph traversal exceeds P99 latency budgets.
• Re-ranking Model: Prevents a slow cross-encoder from blocking the final answer synthesis.
• External API (Tool Calling): Isolates failures from third-party data sources during multi-hop reasoning.

Each dependency gets an isolated breaker to prevent a single failure from taking down the entire retrieval pipeline.

Circuit breaker state transitions are critical signals for agentic observability platforms. Every state change must emit metrics and logs:

• State Gauge: A metric exposing the current state (0=Closed, 1=Open, 2=Half-Open) for real-time dashboards.
• Transition Events: Structured logs capturing the reason for tripping (error rate vs. slow call), the threshold breached, and the duration spent in Open state.
• Failure Counters: Incremented on each rejected request to quantify the blast radius.

This telemetry feeds into Service Level Objective (SLO) monitoring to trigger alerts when error budgets are consumed.

When an approximate nearest neighbor (ANN) index becomes saturated, query latency can spike beyond the Service Level Objective (SLO). A circuit breaker monitors the P99 latency of vector searches and opens when thresholds are breached, immediately returning a semantic cache hit or a graceful degradation response instead of queuing requests that would time out. This prevents connection pool exhaustion and backpressure from propagating to the query understanding layer.

Third-party embedding APIs enforce strict rate limits and can return HTTP 429 errors. A circuit breaker in the semantic indexing pipeline detects this failure mode and transitions to an open state, pausing ingestion rather than retrying aggressively. This avoids retry storms that amplify load and trigger account throttling. During the open state, the system can fall back to a locally cached embedding cache or queue documents for later processing.

In answer synthesis and summarization, a language model endpoint may experience tail latency spikes due to GPU contention or continuous batching overload. A circuit breaker wrapping the inference client detects elevated Time-to-First-Token (TTFT) and opens, routing subsequent requests to a secondary model or a static fallback response. This ensures the user receives a degraded but immediate answer rather than an infinite spinner.

Complex multi-hop reasoning queries can generate expensive SPARQL or Cypher traversals that exceed the Service Level Agreement (SLA) latency budget. A circuit breaker on the graph database connection monitors query duration and opens when a configurable timeout is exceeded, preventing long-running analytical queries from starving the connection pool needed for simple entity lookups. The system can return partial results from previously materialized views.

A cross-encoder re-ranking model may experience performance degradation due to model server overload or hardware faults. A circuit breaker monitors the re-ranker's response time and error rate, opening when failures exceed a threshold. During the open state, the retrieval pipeline bypasses re-ranking entirely and returns results using only the initial Reciprocal Rank Fusion (RRF) scores, maintaining graceful degradation of relevance rather than complete failure.

A cache stampede occurs when a popular semantic cache entry expires, causing dozens of concurrent requests to simultaneously query the vector database for the same embedding. A circuit breaker detects the sudden spike in cache misses and opens, returning a stale cached response while a single background thread refreshes the entry. This pattern, combined with jitter on expiration times, prevents the thundering herd from overwhelming the retrieval backend.

A resilience pattern that isolates system resources into independent pools to prevent a failure in one component from consuming all resources and cascading to others.

• Partitions thread pools, connection pools, or memory limits
• Named after ship compartments that contain hull breaches
• Ensures a memory leak in one module cannot starve the entire application
• Commonly implemented via semaphore-based resource limiting

A transient failure handling strategy that re-attempts a failed request after a delay, with the wait time increasing exponentially between attempts.

• Prevents overwhelming a recovering service with a retry storm
• Parameters: initial delay, multiplier (usually 2x), max attempts, max delay
• Must be idempotent to avoid duplicate side effects
• Often combined with jitter to de-synchronize competing clients

A mechanism that aborts a request if a response is not received within a specified duration, freeing up resources rather than allowing them to hang indefinitely.

• Prevents resource exhaustion from slow or unresponsive dependencies
• Types: connection timeout, read timeout, request timeout
• Too short: false failures; too long: resource waste
• Essential for maintaining tail latency guarantees in distributed systems

An alternative response or behavior returned when a primary dependency fails, allowing the system to gracefully degrade rather than fail completely.

• Examples: cached data, default values, empty lists, or a static placeholder
• Must be semantically appropriate for the calling context
• Often paired with a circuit breaker to serve fallbacks during the open state
• Critical for maintaining partial user experience during outages

A monitoring endpoint or signal that reports the liveness and readiness of a service instance, enabling load balancers and orchestrators to route traffic only to healthy nodes.

• Liveness: Is the process running?
• Readiness: Can it serve traffic (dependencies available)?
• Used by Kubernetes probes and service meshes for automatic failover
• Passive checks observe real traffic; active checks send synthetic requests