Circuit Breaker

An isolation strategy that partitions system resources into independent pools to prevent a failure in one component from consuming all available resources and cascading to other services.

• Resource Isolation: Dedicated thread pools and connection limits per service
• Failure Containment: A saturated model inference pool cannot starve the authorization service
• Implementation: Often achieved through semaphore-based concurrency limits in service meshes

In AI systems, bulkheads ensure that a sudden spike in prompt complexity or a degraded GPU node does not exhaust memory for the entire inference cluster.

A transient error handling mechanism where failed requests are re-attempted with progressively increasing delays, preventing thundering herd retry storms that can overwhelm a recovering service.

• Jitter: Randomization added to backoff intervals to desynchronize competing clients
• Max Retries: A hard limit (often 3-5) before escalating to a circuit breaker trip
• Idempotency Keys: Required to ensure retried inference requests do not produce duplicate side effects

This pattern works in tandem with circuit breakers; the breaker opens only after retries are exhausted.

A design principle where a system continues to operate with reduced functionality rather than failing completely when a dependency is unavailable.

• Fallback Models: Routing requests to a smaller, less capable distilled model when the primary LLM is unreachable
• Static Responses: Serving cached or rule-based outputs for non-critical features
• Feature Flags: Dynamically disabling compute-intensive AI features under load

For example, a customer support chatbot might fall back to keyword-based FAQ retrieval when its generative model circuit breaker is open.

The maximum duration a client will wait for a response before aborting the request. Proper timeout tuning is the first line of defense before a circuit breaker intervenes.

• Connection Timeout: Maximum time to establish a TCP connection (typically 1-5 seconds)
• Read Timeout: Maximum time waiting for the full response body after connection
• LLM-Specific: Inference timeouts must account for variable token generation lengths; a 4096-token output requires significantly more time than a 50-token summary

Timeouts prevent slow resource leaks. A circuit breaker's slow call threshold often triggers before the absolute timeout.

The mechanism by which load balancers, service meshes, and circuit breakers determine if a downstream AI service is capable of accepting traffic.

• Liveness Probe: Answers 'Is the process running?' — a simple HTTP 200 check
• Readiness Probe: Answers 'Can the service accept requests?' — may check GPU memory, model loading status, or database connectivity
• Deep Health Check: Validates end-to-end inference capability by running a synthetic canary prompt through the model

Circuit breakers rely on aggregated health signals. A failing readiness probe increments the failure counter toward the trip threshold.

A traffic shaping control that restricts the number of requests a client can make within a time window, protecting AI APIs from abuse and ensuring fair resource allocation.

• Token Bucket Algorithm: Allows short bursts while enforcing a long-term average rate
• Per-Model Quotas: Different rate limits for GPT-4 versus a lightweight classifier
• 429 Status Code: The standard HTTP response signaling rate limit exceeded, which clients should handle with backoff

While circuit breakers react to failures, rate limiters proactively prevent overload. Both are essential for API gateway configurations in AI platforms.