Circuit Breaker

A circuit breaker operates as a state machine with three distinct states:

• CLOSED: Normal operation. Requests pass through and failures are counted.
• OPEN: The circuit is tripped. Requests fail immediately without attempting the operation.
• HALF-OPEN: A trial state after a timeout. A limited number of requests are allowed to test if the underlying problem is resolved.

The transition from CLOSED to OPEN occurs when a configurable failure threshold (e.g., 5 failures in 60 seconds) is exceeded.

The breaker monitors for specific failures to decide when to trip. Key configurable parameters include:

• Failure Count Threshold: The number of failures (e.g., timeouts, 5xx errors) required to open the circuit.
• Sliding Time Window: The period (e.g., 60 seconds) over which failures are counted.
• Failure Criteria: What constitutes a failure (e.g., HTTP 503, connection timeout, model inference exception).

For AI inference, this could detect repeated failures from a downstream model server or a dependency like a vector database.

When the circuit is OPEN, the system must provide a graceful response instead of cascading failure. Common strategies include:

• Static Fallback: Return a default, cached, or simplified response.
• Stale Data: Serve slightly outdated data from a cache.
• Queueing/Delayed Processing: Place requests in a queue for later retry.
• Alternative Service Path: Route the request to a backup or degraded service instance.

This prevents user-facing timeouts and allows the overloaded service time to recover.

After a configured reset timeout, the circuit moves to HALF-OPEN. This state is critical for automatic recovery:

• A limited number of trial requests are allowed to pass.
• If these requests succeed, the circuit assumes the fault is resolved and moves back to CLOSED.
• If they fail, the circuit returns to OPEN for another reset timeout period.

This probe mechanism prevents a recovered service from being immediately flooded by a backlog of requests.

Circuit breakers are a primary source of operational telemetry. They should emit:

• Metrics: State transition counts (closed-to-open), failure rates, request volumes.
• Logs: Events for each state change with contextual metadata (e.g., error cause).
• Traces: Annotate distributed traces to show when a request was short-circuited.

This data is essential for SLO/SLI monitoring and understanding systemic fragility. Tools like Prometheus and Grafana visualize these metrics.

In Production PEFT Servers, a circuit breaker protects the system from:

• Downstream Model Server Failure: If the Triton or vLLM server becomes unresponsive.
• Adapter Loading Latency: Excessive delay when switching adapters in a multi-adapter serving setup.
• Cache Stampedes: Thundering herd problems when a cold model starts.

By tripping the circuit, the system can fail fast, return a fallback response, and prevent thread pool exhaustion in the calling application, which is crucial for maintaining overall system latency.

A circuit breaker guards your application when calling external model APIs (e.g., OpenAI, Anthropic) or internal microservices. When the downstream service experiences high latency or errors, the breaker trips open, failing requests immediately instead of letting them timeout. This prevents thread pool exhaustion in your application. The breaker periodically allows a test request (half-open state) to probe for recovery before closing to resume normal traffic.

• Example: An LLM-powered chatbot stops calling a failing sentiment analysis service, returning a graceful fallback message instead of hanging.

Circuit breakers can be triggered by model performance metrics instead of just HTTP errors. By monitoring live metrics like prediction latency, throughput, or business KPIs (e.g., click-through rate), a breaker can trip when a deployed model begins to degrade due to concept drift or data pipeline issues.

• Implementation: Integrate with your ML observability platform. If the 95th percentile latency for a vision model spikes from 100ms to 2s, the breaker trips, routing traffic to a stable fallback model or version.

In production PEFT servers hosting multiple adapters or LoRA weights, a circuit breaker prevents a single tenant or task from monopolizing GPU memory and compute. It monitors hardware metrics like GPU memory utilization, SM (Streaming Multiprocessor) activity, and inference queue depth.

• Use Case: A misbehaving client sending extremely long sequences could cause out-of-memory (OOM) errors for all users. A breaker on the dynamic batching layer trips, rejecting that client's requests to protect overall service stability.

In continuous model learning systems, a faulty feedback collection or online learning pipeline can create a destructive loop. A circuit breaker sits between the live inference service and the feedback logging/retraining pipeline. If the data pipeline is backlogged or producing corrupt data, the breaker opens, temporarily halting feedback ingestion.

• Prevents: Poisoned training data from a broken user interface from triggering an unnecessary and potentially harmful automated retraining job.

In a multi-tenant inference service where different clients or tasks use different adapter modules, a circuit breaker provides fault isolation. A bug or malformed input specific to one tenant's adapter (e.g., a specific LoRA weight) should not crash the server for all tenants.

• Implementation: Implement tenant-specific breakers on the adapter switching layer. If loading a particular adapter consistently times out, that tenant's circuit opens, returning an error, while other tenants continue unaffected.

Circuit breakers are a critical safety mechanism during canary deployments of new model versions. As a small percentage of traffic is routed to the new canary, its circuit breaker is configured with aggressive thresholds. Any sign of instability—increased error rates, novel exceptions, or performance regression—causes the canary's breaker to trip instantly.

• Result: Traffic is automatically rerouted back to the stable version, containing the blast radius of a bad deployment without requiring manual intervention.

A safe deployment strategy where a new model version processes live inference requests in parallel with the production model, but its predictions are logged and not returned to users. This allows for performance comparison (e.g., accuracy, latency) and validation against the live data distribution without any risk to the user experience. It is often used before a canary deployment.

• Key Use Case: Evaluating a new LoRA adapter or fine-tuned model against the current champion model using real-time traffic.

A periodic probe (e.g., an HTTP endpoint /health) that a container orchestrator (like Kubernetes) or load balancer uses to determine if a service instance is operational. For an inference server, a health check might verify that the base model and adapters are loaded, the GPU is accessible, and the service can perform a dummy inference. Failed health checks trigger automatic instance restart or removal from the load balancer pool.

• Liveness vs. Readiness: Liveness probes check if the container is running; Readiness probes check if it can accept traffic.

The property of an API operation whereby performing it multiple times has the same effect as performing it once. For inference endpoints, designing idempotent POST requests (often using a client-supplied request ID) is critical for safe retries. If a client times out and retries a request, the circuit breaker pattern may be involved in failing fast, but idempotency ensures the retry doesn't cause duplicate side effects (e.g., logging the same prediction twice for billing).

• Circuit Breaker Context: When a circuit breaker is open and failing requests fast, clients will retry; idempotency prevents retries from causing data corruption.

An inference architecture where a single base model instance can dynamically load and switch between multiple trained adapter modules or LoRA weights to handle different tasks or tenants without restarting. This requires sophisticated routing and adapter switching logic. A circuit breaker in this context might be applied per-adapter or per-tenant route to isolate failures—if one adapter's logic fails repeatedly, it can be tripped without affecting requests to other adapters on the same server.

• Isolation: Enables fault isolation between different fine-tuned tasks served from a shared infrastructure.