Circuit Breaker

A circuit breaker operates as a finite state machine with three distinct states:

• Closed: Requests flow normally. A failure counter tracks consecutive errors. When the counter exceeds a threshold, the breaker transitions to Open.
• Open: Requests are immediately rejected without attempting the call. This fast-fails to prevent resource exhaustion. A timer is started for a predefined cooldown period.
• Half-Open: After the cooldown expires, a limited number of probe requests are allowed through. If they succeed, the breaker resets to Closed. If they fail, it reverts to Open and resets the timer.

The transition from Closed to Open is governed by a failure threshold. This is not simply a count of exceptions; it must be tuned to the specific failure modes of AI services.

• Consecutive Failures: The simplest counter. Resets on a single success.
• Error Rate over Time Window: A sliding window calculates the percentage of failed requests. This is more robust for intermittent issues like GPU out-of-memory errors.
• Slow Call Threshold: Treats requests exceeding a latency percentile (e.g., p99 > 5s) as failures, preventing resource saturation from stalled model inference.

The cooldown period is the time the breaker remains in the Open state before transitioning to Half-Open. This duration must exceed the downstream service's typical recovery time.

• Exponential Backoff Cooldown: The cooldown duration increases with each successive trip (e.g., 10s, 30s, 90s) to avoid thrashing a persistently degraded model.
• Probe Request Limiting: In the Half-Open state, only a configurable number of requests per time unit are permitted. If the probe succeeds, the breaker closes. This prevents a flood of traffic from re-overloading a just-recovered model serving endpoint.

When a circuit breaker is Open and rejects a request, the system must not simply return an error to the user. A fallback strategy provides graceful degradation.

• Static Response: Return a cached, pre-computed, or default response. For a recommendation model, this might be a curated list of popular items.
• Stale Data: Serve the last known good result from a local cache, even if it's expired. This is acceptable for non-critical personalization features.
• Alternative Service: Call a simpler, more robust model or a rule-based heuristic. A complex large language model could fall back to a deterministic template-based response.

Circuit breakers should not operate in isolation. They must integrate with the service's health check endpoints to enable informed decision-making.

• Deep Health Checks: A standard liveness check might return 200 OK while the model's inference queue is saturated. A deep health check probes the actual inference path and reports degraded status, which can be used to proactively trip the breaker.
• Orchestration Awareness: In a Kubernetes environment, a breaker can observe pod readiness states. If all pods for a model deployment are failing readiness probes, the breaker can open without waiting for its own failure counter to reach the threshold.

The granularity of the circuit breaker is a critical design decision that directly impacts blast radius.

• Per-Service Breaker: A single breaker protects all calls to a model serving endpoint. A failure in one model version trips the breaker for all models on that host. This is simple but has a large blast radius.
• Per-Model Breaker: A dedicated breaker instance is created for each unique model version or endpoint path. A failure in the v2-experimental model does not affect traffic to the stable v1 model. This aligns with bulkhead isolation principles and is the recommended pattern for multi-model serving platforms.