Graceful Degradation

A stability pattern that automatically stops requests to a failing AI service to prevent cascading failures. When error rates exceed a threshold, the breaker trips to a fail-fast state, immediately rejecting requests rather than allowing them to queue and time out. This protects upstream services from resource exhaustion while the downstream component recovers. Circuit breakers typically implement three states: closed (normal operation), open (requests rejected), and half-open (limited trial requests to test recovery).

A resilience pattern that partitions AI serving resources into isolated pools to prevent a failure in one model or tenant from consuming all available system capacity. Named after ship compartments that contain hull breaches, bulkheads allocate dedicated thread pools, connection limits, and memory quotas per service. If one model experiences latency degradation or resource exhaustion, other models continue operating normally. This is critical for multi-tenant AI platforms where noisy-neighbor problems can cascade.

The automatic switching to a redundant or standby AI instance upon detecting a failure in the primary production system. Failover mechanisms rely on continuous health checks to detect unresponsive endpoints and route traffic to hot, warm, or cold standbys. In AI systems, failover may involve switching to a simpler heuristic model when the primary deep learning model becomes unavailable—a direct implementation of graceful degradation. Recovery Time Objective (RTO) defines the maximum acceptable failover duration.

A deliberate strategy of dropping a portion of incoming AI inference requests during overload to preserve acceptable latency for the remaining traffic. Unlike circuit breakers that block all requests, load shedding applies selective throttling based on request priority, client tier, or resource cost. High-priority requests continue processing while low-priority or expensive requests receive immediate rejection responses. This prevents the system from entering a death spiral where queued requests consume all memory and crash the service entirely.

A retry strategy that progressively increases the wait time between consecutive failed requests to an AI service. When a client receives an error response, it waits for an initial interval before retrying; each subsequent failure doubles the wait period. This prevents thundering herd problems where many clients simultaneously retry and overwhelm a recovering service. Combined with jitter (randomized delay), exponential backoff distributes retry attempts across time, giving downstream AI components space to stabilize.

A persistent storage buffer for AI inference requests that cannot be processed despite retries, enabling offline debugging and preventing data loss. When a request exhausts its retry policy or encounters a non-transient error, it moves to the DLQ rather than being silently discarded. Operations teams can inspect queued messages to identify systemic failure patterns, reprocess corrected requests, or extract audit data. This is essential for maintaining decision provenance in regulated AI workflows.