Circuit Breaker

A circuit breaker operates as a deterministic state machine with three distinct phases:

• CLOSED: Requests flow normally to the model endpoint while a failure counter tracks consecutive errors.
• OPEN: After the failure threshold is breached, the breaker trips and immediately rejects all requests with a fallback response, preventing further load on the failing service.
• HALF-OPEN: After a configurable cooldown period, a limited number of probe requests are permitted to test if the downstream model has recovered. Success resets the breaker to CLOSED; failure returns it to OPEN.

The sensitivity of a circuit breaker is governed by two critical parameters that must be tuned to the specific model endpoint:

• Consecutive Failure Count: The number of sequential failures required to trip the breaker. A value of 5 is common for critical inference paths.
• Error Rate Percentage: An alternative threshold based on the ratio of failures to total requests within a sliding time window, often set at 50% over a 30-second window.
• Slow Call Threshold: Requests exceeding a defined latency percentile (e.g., P99 > 500ms) can be classified as failures to protect against degraded but not dead endpoints.

When a circuit breaker is OPEN, the system must return a degraded but deterministic response rather than propagating an error to the end user:

• Static Recommendations: Return a pre-computed, non-personalized top-N product list from a cache.
• Stale Cache Serving: Serve the last known good prediction from a local in-memory cache with a short TTL.
• Default Embedding: Substitute a global average user embedding when the personalization model is unreachable.
• Empty Result Set: For non-critical UI components, return an empty array with a degraded: true flag in the response envelope.

Circuit breakers are most effective when combined with the Bulkhead pattern, which partitions thread pools and connection pools per downstream dependency:

• A circuit breaker scoped to the 'recommendation model' thread pool prevents its saturation from starving the 'search ranking' pool.
• In Kubernetes, this maps to dedicated sidecar proxies (e.g., Envoy or Linkerd) that enforce per-cluster circuit breaking at the service mesh layer.
• Without bulkheading, a single tripped breaker can still cause resource contention if all services share a common, unbounded thread pool.

Every state transition in a circuit breaker must emit metrics and structured log events for operational visibility:

• Key Metrics: circuit_breaker_state (gauge: 0=CLOSED, 1=OPEN, 2=HALF_OPEN), circuit_breaker_trip_total (counter), and circuit_breaker_fallback_total (counter).
• Alerting Rules: An alert should fire when a breaker remains OPEN for longer than the expected recovery window (e.g., 5 minutes), indicating a persistent downstream outage.
• Distributed Tracing: Inject the breaker state into trace span tags to correlate user-facing latency spikes with specific circuit trips.

Circuit breaking can be implemented at different architectural scopes:

• Per-Node (Sidecar): Each model serving replica tracks its own failure counters independently. This is simple but can lead to partial degradation where some replicas trip while others remain healthy.
• Distributed Consensus: A centralized state store (e.g., Redis or etcd) aggregates failure counts across all replicas, ensuring a consistent breaker state. This prevents 'flapping' but introduces a dependency on the consensus store itself.
• Hybrid Approach: Use per-node breakers for fast local decisions with a distributed backplane for coordinated HALF-OPEN probing, balancing autonomy with consistency.

A defensive resilience strategy where a server intentionally drops a fraction of incoming requests when it detects overload, prioritizing the successful processing of remaining traffic over total failure.

• Triggered by queue depth thresholds or latency degradation
• Drops requests at the edge (load balancer) or application layer
• Often paired with priority queues: shed low-priority requests first
• Example: Drop non-critical recommendation requests while preserving checkout API calls during a traffic spike

A control mechanism that restricts the number of inference requests a client can make within a specific time window, protecting model serving infrastructure from abuse and accidental overload.

• Token bucket and leaky bucket are common algorithms
• Enforced at the API gateway or service mesh layer
• Prevents a single noisy tenant from starving shared model endpoints
• Distinct from circuit breaking: rate limiting is proactive throttling, circuit breaking is reactive isolation

The design principle of maintaining reduced but functional service when dependencies fail. When a circuit breaker trips and blocks calls to a primary model, the system falls back to a degraded but acceptable alternative.

• Fallback strategies: cached predictions, simpler heuristic models, static recommendations
• Static fallback: serve pre-computed popular items when personalization model is unavailable
• Stale cache: return last-known-good embeddings while retraining pipeline recovers
• Measures degradation with degraded-mode SLOs that are lower than normal targets

A resource isolation pattern that partitions serving infrastructure into separate pools, so a failure in one pool cannot exhaust resources needed by another. Named after ship compartments that prevent a single hull breach from sinking the vessel.

• Thread pool isolation: dedicate separate thread pools per model or client
• Connection pool isolation: separate connection pools per downstream dependency
• Prevents a noisy neighbor from consuming all memory or connections
• Works with circuit breakers: bulkheads contain the blast radius, circuit breakers stop the calls

A resilience pattern where failed requests are retried with progressively increasing delays between attempts, preventing thundering herd retry storms that can overwhelm recovering services.

• Jitter: random variation added to backoff intervals to desynchronize retry waves
• Max retries: circuit breakers often count retries toward failure thresholds
• Idempotency: retries require idempotent operations to avoid duplicate side effects
• Example: Retry at 100ms, 200ms, 400ms, 800ms with ±25% jitter before opening the circuit