Circuit Breaker

A circuit breaker operates through a finite state machine with three core states:

• Closed: The normal operating state. Requests flow to the service. Failures are counted.
• Open: The tripped state. All requests fail immediately without calling the service, returning a pre-defined fallback or error.
• Half-Open: A probationary state. After a timeout, a single test request is allowed. Its success resets the breaker to Closed; its failure returns it to Open.

The breaker transitions from Closed to Open based on configurable thresholds that detect a failing dependency.

• Failure Count/Percentage: A sliding window tracks recent request outcomes. The breaker trips when failures exceed a set count (e.g., 5 failures) or a percentage (e.g., 50% failure rate).
• Timeout Detection: Individual calls are wrapped with a timeout. A slow or unresponsive service that exceeds this duration is counted as a failure, protecting the caller from latency spikes.

When the breaker is Open, calls must not reach the unhealthy service. Instead, the pattern mandates a fallback strategy to maintain partial functionality.

• Static Response: Return cached data, a default value, or a generic "service unavailable" message.
• Alternative Service: Route the request to a backup or degraded service tier.
• Fast Failure: Immediately throw an exception to the caller, which is preferable to letting threads pool indefinitely waiting for a timeout. This allows the caller's own logic to handle the failure.

The circuit breaker does not stay open indefinitely. After a configured reset timeout, it moves to the Half-Open state.

• A single probe request is sent to the failing service.
• Success: The breaker assumes the service has recovered, resets the failure count, and transitions back to Closed.
• Failure: The breaker returns to the Open state, and the reset timer starts again. This prevents a recovering but still unstable service from being flooded immediately.

Circuit breakers and retries are complementary but distinct patterns that must be coordinated to avoid conflict.

• Retry Logic handles transient faults (e.g., a momentary network glitch). It operates within the Closed state of the circuit breaker.
• Circuit Breaker handles persistent faults (e.g., a service is down). It supersedes retry logic. When the breaker is Open, no retries should be attempted, as they would be guaranteed to fail and waste resources. The combination is often called "retry with circuit breaker."

The state of circuit breakers is a vital health metric for a distributed system and must be exposed for monitoring.

• Metrics: Count of state transitions (Closed → Open, Open → Half-Open), failure rates, and request volumes per state.
• Logging & Tracing: Log state changes with contextual information (service name, failure count). Propagate the breaker state in distributed traces (e.g., as a tag in OpenTelemetry).
• Dashboards & Alerts: Visualize breaker status across services. Alert engineering teams when a breaker trips (Open state), as it indicates a significant downstream failure.

When an LLM application calls an external model API (e.g., OpenAI, Anthropic), a circuit breaker monitors for failures like timeouts, high latency, or quota errors. After a threshold of failures, it opens the circuit, failing fast for subsequent requests. This prevents the application from exhausting resources or degrading while waiting for a non-responsive service. The breaker periodically allows a test request (half-open state) to check if the API has recovered before closing and resuming normal traffic.

During canary deployments or A/B testing of new model versions, a circuit breaker can be placed in front of the experimental endpoint. It tracks error rates (e.g., from output validation systems) or performance degradation (e.g., latency spikes). If the new model's error rate exceeds a defined Service Level Objective (SLO), the breaker opens, automatically routing all traffic back to the stable version. This enables safe, automated rollback without manual intervention.

In a RAG pipeline, the circuit breaker protects the application from failures in its knowledge retrieval components.

• Vector Database Failures: If the semantic search to a vector database times out or returns errors, the breaker can open, allowing the LLM to fall back to its parametric knowledge or return a graceful degradation message.
• External Data Source Failures: For pipelines that query live APIs or databases for grounding data, a breaker prevents the LLM from stalling or producing incomplete answers when these sources are unavailable.

Circuit breakers enforce budget guards and prevent resource exhaustion in multi-tenant LLM platforms.

• Token Budget Enforcement: A breaker can track cumulative token usage per user/session. If usage exceeds a pre-defined budget within a time window, the circuit opens, blocking further generation to control API costs.
• GPU Memory Protection: For self-hosted models, a breaker can monitor GPU memory utilization. If an inference request pattern risks causing an Out-Of-Memory (OOM) error, the breaker opens to reject new requests, allowing the system to clear its queue and avoid a full pod crash.

In a microservices architecture for AI features, circuit breakers are often implemented at the service mesh layer (e.g., Istio, Linkerd).

• The mesh monitors traffic between services (e.g., between a frontend and a model-serving service).
• It automatically opens circuits based on configurable thresholds for HTTP error codes (5xx) or latency percentiles.
• These events are fed into the observability stack (metrics, logs, traces), providing a unified view of system resilience and triggering alerts for Site Reliability Engineering (SRE) teams.

In multi-agent systems or complex agentic cognitive architectures, a single failing tool call or agent can stall an entire reasoning loop. Circuit breakers are applied to individual agent actions or tool executions.

• If an agent repeatedly fails to call a specific external API (e.g., for weather data), its circuit for that tool opens.
• The agent's recursive error correction logic can then trigger, allowing it to replan its approach, select an alternative tool, or escalate the task within the orchestration framework, maintaining overall workflow progress.

The Circuit Breaker pattern is often implemented alongside Retry Logic. When a request fails, the system may retry it. However, to prevent overwhelming a failing service, Exponential Backoff is used: the wait time between retries increases exponentially (e.g., 1s, 2s, 4s, 8s). The circuit breaker monitors these failures; if they exceed a threshold, it 'trips' to open state, suspending all retries for a period.

While a Circuit Breaker protects a client from a failing downstream service, Rate Limiting protects a service from excessive requests from an upstream client. Both are flow-control mechanisms:

• Rate Limiter: "You (client) are sending too many requests too fast."
• Circuit Breaker: "The service I'm calling is failing, so I'll stop asking for a while." They are complementary and often used together on API Gateways to enforce global policies.

Circuit Breakers rely on failure detection, which is closely related to health monitoring. In Kubernetes, this is formalized with probes:

• Liveness Probe: Determines if a container needs a restart.
• Readiness Probe: Determines if a container can accept traffic. A circuit breaker acts as a dynamic, application-layer health check for remote services. If a downstream service's readiness probe fails, Kubernetes stops routing traffic to it; if its API calls fail, the client's circuit breaker opens.

Load Balancers distribute traffic across healthy backend instances. A Circuit Breaker is a client-side load management pattern. In modern microservices, these concepts converge within a Service Mesh (e.g., Istio, Linkerd). The mesh's sidecar proxies implement circuit breaking, retries, and load balancing at the network layer, decoupling resilience logic from application code. The mesh configures failure thresholds (e.g., consecutive 5xx errors) that trigger circuit breaking for a specific host.

Chaos Engineering is the practice of intentionally injecting failures (e.g., latency, errors) into a system to test its resilience. Circuit Breakers are a primary defense mechanism that such experiments validate. A chaos experiment might:

• Inject 100% HTTP 500 errors into a payment service.
• Validate that the checkout service's circuit breaker trips correctly.
• Confirm that the failure is contained and the overall system remains functional. This builds confidence that the circuit breaker configuration is effective in production.

Circuit Breakers are critical for safe deployment strategies. In a Canary Deployment, a new version is released to a small subset of traffic. A misbehaving canary could cause errors for those users. A well-configured circuit breaker in the calling services will trip, isolating the failure and protecting the broader system. In Blue-Green Deployment, traffic is switched entirely from one environment to another. Circuit breakers help ensure the new 'green' environment is healthy before and after the switch, providing a fast automatic rollback signal if it begins to fail.