Circuit Breaker

A circuit breaker operates as a deterministic finite-state machine with three distinct states:

• Closed: Requests flow normally. A failure counter tracks consecutive errors.
• Open: The breaker trips after a failure threshold is exceeded. All requests are immediately rejected with a CircuitBreakerOpenException, failing fast without invoking the downstream service.
• Half-Open: After a cooldown period, a limited number of probe requests are allowed through. If they succeed, the breaker resets to Closed; if they fail, it reverts to Open.

This prevents thundering herd problems during recovery.

The sensitivity of a circuit breaker is governed by tunable parameters that define what constitutes a failure:

• Consecutive Failure Count: The number of sequential failures required to trip the breaker (e.g., 5).
• Failure Rate Percentage: A sliding window metric where the breaker trips if the error rate exceeds a configured percentage (e.g., >50% of requests in the last 30 seconds).
• Slow Call Threshold: Requests exceeding a defined latency percentile (e.g., 99th percentile > 2 seconds) can be classified as failures to protect against resource saturation.

These thresholds must be calibrated to the specific Service Level Objective (SLO) of the downstream dependency.

When a circuit is Open, the breaker must provide a defined fallback strategy to maintain upstream stability:

• Fail-Fast: Immediately returns an error response, allowing the caller to handle the exception gracefully without thread pool exhaustion.
• Fallback Response: Returns a cached, default, or degraded response (a stale-while-revalidate pattern) to provide partial functionality.
• Silent Failure: Logs the error but returns a null or empty response, suitable for non-critical advisory services.

This prevents a single slow downstream service from consuming all available worker threads in the upstream application, a failure mode known as resource exhaustion.

The transition from Open to Half-Open is a critical recovery mechanism that must be carefully managed:

• Cooldown Timer: A fixed duration (e.g., 30 seconds) the breaker remains Open before allowing probes. This gives the downstream service time to recover from load spikes or transient failures.
• Max Requests in Half-Open: A configurable limit on the number of concurrent probe requests (often set to 1) to prevent overwhelming a recovering service.
• Success Threshold: The number of consecutive successful probes required to fully close the circuit and resume normal operation.

Without a probing strategy, the system risks flapping—oscillating rapidly between Open and Closed states.

Circuit breakers must emit rich telemetry data to enable operational visibility:

• State Change Events: Every transition between Closed, Open, and Half-Open must be logged and emitted as a metric for alerting.
• Failure Counters: Incremented counters for each failure type (timeout, HTTP 5xx, connection refused) to diagnose root causes.
• Bulkhead Correlation: Circuit breaker metrics are often correlated with bulkhead thread pool saturation to distinguish between downstream failure and local resource exhaustion.
• Distributed Tracing: The breaker's decision (allow/deny) should be attached to span tags in systems like OpenTelemetry to trace the blast radius of a failure.

For a circuit breaker to be safe, the operations it guards must be idempotent where possible:

• Retry Risk: If a timeout triggers the breaker but the downstream request actually succeeded, a naive retry could cause duplicate mutations (e.g., double-charging a payment).
• Idempotency Keys: Upstream callers must pass a unique key to ensure that retried requests are recognized and deduplicated by the downstream service.
• Safe Verbs: Circuit breakers are safest on idempotent HTTP methods (GET, PUT, DELETE). POST operations require explicit idempotency handling.

Without idempotency, the fail-fast behavior of a circuit breaker can introduce data integrity issues.

When a language model provider throttles requests, a circuit breaker opens immediately to prevent queue buildup. Instead of retrying and compounding the problem, the system returns a cached or fallback response.

• Half-open state probes the endpoint every 30 seconds
• Prevents thundering herd retry storms across microservices
• Logs 429 Too Many Requests as a metric for capacity planning

Example: A content generation pipeline hitting OpenAI's 3,500 RPM limit trips the breaker, serving stale-but-accurate product descriptions from cache until the window resets.

Semantic search retrieval depends on vector database availability. A circuit breaker monitors p99 latency and error rates on embedding lookups, opening when the database enters degraded mode.

• Falls back to keyword-based BM25 retrieval when vector search fails
• Prevents request threads from blocking indefinitely on slow index queries
• Monitors connection pool exhaustion as a leading indicator

This ensures retrieval-augmented generation pipelines maintain partial functionality rather than failing completely during database incidents.

Automated factuality checks and entailment verification services are critical path components. When a validation endpoint becomes unresponsive, a circuit breaker isolates the failure.

• Routes content through a degraded publishing mode with human-review flags
• Prevents the entire content pipeline from stalling on a single microservice
• Implements exponential backoff with jitter for recovery attempts

This pattern is essential for high-throughput programmatic SEO systems where a single downstream failure could halt thousands of page generations.

Retrieval-augmented generation systems query enterprise knowledge graphs and document stores. A circuit breaker wraps each external data source independently, preventing one slow knowledge base from degrading all content generation.

• Each data source gets its own isolated breaker configuration
• Timeout thresholds tuned to p95 latency of each source
• Failed queries return partial results with provenance warnings

This bulkhead-like isolation ensures that a SharePoint outage doesn't block content that relies on Confluence or PostgreSQL-backed knowledge stores.

When GPU inference queues exceed capacity, a circuit breaker implements load shedding by rejecting low-priority generation requests. This preserves resources for critical, latency-sensitive content tasks.

• Assigns priority tiers to content generation jobs
• Low-priority batch content gracefully degrades to retry-later status
• Prevents head-of-line blocking where one large request starves others

This pattern is critical for real-time personalization engines where sub-second response times are required for user-facing content assembly.