Circuit Breaker

A circuit breaker operates as a finite state machine with three distinct states:

• Closed: Requests flow normally to the service. Failures are counted.
• Open: The circuit 'trips.' All requests fail immediately without calling the service, allowing it time to recover.
• Half-Open: After a timeout, a limited number of test requests are allowed. Success resets the circuit to Closed; failure returns it to Open.

The breaker monitors requests for failures (e.g., timeouts, HTTP 5xx errors). It trips based on configurable thresholds:

• Failure Count: Trip after N consecutive failures.
• Failure Ratio: Trip when the percentage of failed requests exceeds a set limit within a sliding time window.
• Timeout Duration: Define a maximum wait time for a request before it's considered a failure.

This is the self-healing mechanism. After a configured reset timeout in the Open state, the breaker transitions to Half-Open. A single request or a small batch is permitted. Its success indicates the service may be healthy, resetting the breaker to Closed. Its failure immediately returns the breaker to Open, restarting the timeout. This prevents a recovering service from being flooded.

When the circuit is Open, calls fail immediately (often with a predefined exception like CircuitBreakerOpenError). This fail-fast behavior:

• Reduces latency for users (no waiting for timeouts).
• Prevents thread pool exhaustion in the calling service.
• Should be paired with a fallback strategy, such as returning a cached value, a default response, or delegating to a more robust fallback model.

In ML systems, a circuit breaker is typically implemented at the API gateway or service mesh layer protecting inference endpoints. It guards against:

• Model server crashes or restarts.
• GPU memory exhaustion causing OOM errors.
• Extreme latency spikes from model inference.
• Downstream dependency failures (e.g., a feature store). It is a key component for meeting Service Level Objectives (SLOs) for availability and latency.

A circuit breaker and a retry mechanism are complementary but distinct patterns:

• Retry: Attempts the same failing request again, useful for transient faults (e.g., network blip).
• Circuit Breaker: Stops all requests to prevent overloading a failing service. It is a systemic response to persistent failure. Best practice is to use retries with exponential backoff inside a Closed circuit. If retries consistently fail, the breaker will trip, moving the system to a different mitigation strategy.

When a model endpoint begins to exhibit high latency or error rates—often due to concept drift, data anomalies, or resource exhaustion—a circuit breaker trips to stop sending requests. This prevents backpressure from cascading to upstream services like web servers or mobile apps. The system can then invoke a fallback model or return a cached/default response until the primary model recovers, as monitored by health checks.

In complex architectures like Retrieval-Augmented Generation (RAG) or agentic systems with tool calling, a circuit breaker isolates failures in critical dependencies.

• Example: If a vector database query times out, the circuit breaker on the retrieval component opens. The system can then fall back to a keyword search or proceed with limited context, preventing the entire LLM pipeline from stalling. This is essential for maintaining response time SLOs.

When integrating with external model APIs (e.g., OpenAI, Anthropic), circuit breakers protect against budget overruns and quota exhaustion. If the external service returns repeated errors or severe latency spikes, the breaker trips. Requests are then routed to a cheaper, local small language model or a cached response. This pattern is a core component of inference optimization strategies for managing latency and cost predictability.

During canary releases or blue-green deployments of new model versions, circuit breakers are deployed per version. If the canary's error rate exceeds a threshold (e.g., 5%), its breaker trips instantly, diverting all traffic back to the stable version. This enables automated rollback based on real-time metrics, a key practice in progressive delivery and MLOps pipelines. It moves beyond simple traffic splitting to active failure containment.

In large-scale model serving clusters using NPUs/GPUs, individual accelerators can fail. A circuit breaker at the node or pod level detects failures via failed health checks or kernel errors. The load balancer stops routing requests to the faulty node, allowing the autoscaling system to provision a replacement. This maintains aggregate throughput and is vital for meeting availability SLOs in immutable infrastructure.

A circuit breaker operates in three states:

• CLOSED: Requests flow normally.
• OPEN: Requests fail fast; no calls are made to the failing service.
• HALF-OPEN: After a timeout, a trial request is sent. Success closes the circuit; failure resets the open timer.

Key Configurations:

• Failure Threshold: Error percentage to trip (e.g., 50%).
• Timeout Duration: How long to stay open before testing recovery.
• Sliding Window: The time window for calculating the error rate.

These are often configured in service mesh (e.g., Istio) or client-side libraries (e.g., resilience4j).

A deployment strategy where a new model version is initially exposed to a small, specific subset of users or infrastructure. This allows for real-world validation of performance and stability with minimal risk before a full rollout. It is often used in conjunction with a circuit breaker; if the canary fails, the circuit breaker can isolate it, and traffic is routed back to the stable version.

• Key Mechanism: Controlled exposure to a limited segment.
• Primary Goal: Validate new versions safely in production.
• Relation to Circuit Breaker: The canary is the 'service' the circuit breaker protects. If the canary endpoint fails, the breaker trips to prevent user impact.

A deployment technique where a new model processes live production traffic in parallel with the currently serving model, but its predictions are logged and not used to affect user-facing decisions. This allows for extensive performance comparison and validation without risk. A circuit breaker is less critical here as the shadow model is not in the critical path, but the pattern is part of the same safety-first philosophy.

• Key Mechanism: Parallel, non-critical execution on live data.
• Primary Goal: Gather performance data and validate predictions safely.
• Safety Layer: Sits alongside canary releases and circuit breakers in a progressive delivery strategy.

A simpler, more robust model (e.g., a heuristic, rule-based system, or previous stable version) that is invoked when the primary model fails, times out, or produces low-confidence predictions. The circuit breaker pattern is the mechanism that triggers this switch. When the breaker trips on the primary model endpoint, traffic is automatically rerouted to the fallback to ensure continuous service availability.

• Key Mechanism: Provides a reliable backup prediction source.
• Primary Goal: Guarantee system uptime and a baseline user experience during failures.
• Critical Dependency: Enabled by the circuit breaker's fast-fail logic.

A periodic probe (e.g., an HTTP request) sent to a service, like a model inference endpoint, to verify its operational status and readiness to handle traffic. Health checks are a fundamental input to a circuit breaker's state machine. A series of failed health checks can be the signal that causes the circuit to transition from CLOSED to OPEN, proactively stopping traffic to a failing service.

• Key Mechanism: Active polling for liveness and readiness.
• Primary Goal: Determine service availability before sending real user requests.
• Integration Point: Often the primary metric a circuit breaker monitors to make its trip decision.

An emergency mechanism, often implemented as a feature flag or configuration setting, that allows operators to instantly and manually disable a specific feature or revert to a fallback. While a circuit breaker is automated based on failure metrics, a kill switch is a manual override. They are complementary: a kill switch can be pulled if automated monitoring (or a circuit breaker) hasn't yet triggered but an issue is identified.

• Key Mechanism: Manual, immediate deactivation.
• Primary Goal: Provide human-in-the-loop control for catastrophic failures.
• Relation to Circuit Breaker: The kill switch is the manual counterpart to the automated circuit breaker.

The practice of routing a controlled percentage of incoming inference requests to different model versions or endpoints. This is controlled by a load balancer or service mesh and is essential for A/B testing and gradual rollouts. A circuit breaker operates downstream of traffic splitting. If traffic is split 90/10 between a champion and challenger model, a circuit breaker would protect each individual endpoint, ensuring a failure in the challenger doesn't affect the champion's traffic.

• Key Mechanism: Distributing request load across multiple backends.
• Primary Goal: Enable comparative testing and phased deployments.
• Defensive Layer: Circuit breakers add resilience to each backend service receiving split traffic.