Load Shedding

Load shedding is a proactive control mechanism, distinct from reactive failure handling. It is triggered by predictive metrics (e.g., queue depth, system load) before resources are fully exhausted and errors cascade. This contrasts with patterns like retries or fallbacks, which activate after a failure has occurred. The goal is to prevent a total system collapse by intentionally sacrificing some functionality to preserve core operations.

Effective load shedding requires a classification system for incoming requests. Traffic is typically categorized by:

• Criticality: Mission-critical API calls vs. background or batch jobs.
• Resource Cost: High-latency database queries vs. simple cache lookups.
• User Impact: Actions affecting real-time transactions vs. non-essential features. Systems use this classification to define shedding policies, dropping low-priority requests first while maintaining a quality of service (QoS) guarantee for high-priority traffic.

Load shedding and circuit breakers are complementary patterns within a resilience strategy. A circuit breaker protects a client from calling a failing downstream service, while load shedding protects a server from being overwhelmed by upstream clients. They are often used together:

• A service under load may shed its own non-critical traffic.
• Simultaneously, its downstream dependencies may have their circuit breakers open, causing failures that further inform the shedding service's health metrics. This creates a layered defense against cascading failures.

Shedding decisions are based on real-time system metrics, not arbitrary thresholds. Common triggers include:

• System Load: CPU, memory, or I/O utilization exceeding a defined ceiling (e.g., >85%).
• Queue Depth: The number of pending requests in an application or thread pool queue.
• Latency Percentiles: P95 or P99 response times degrading beyond a Service Level Objective (SLO).
• Concurrent Connections: The number of active HTTP/gRPC connections approaching a limit. These metrics are monitored over a rolling time window to avoid reacting to transient spikes.

The objective is graceful degradation, not abrupt failure. Implementations should:

• Return a clear, non-retryable error (e.g., HTTP 503 Service Unavailable with a Retry-After header) to prevent client retries from exacerbating the load.
• Provide actionable logging and observability to distinguish shed traffic from genuine errors.
• Where possible, queue or defer low-priority work instead of outright rejection. This maintains user trust by communicating the system's state transparently and preserving functionality for the most important workflows.

Advanced systems employ adaptive load shedding, where shedding policies and thresholds adjust dynamically based on:

• Time of Day or Traffic Patterns: Stricter thresholds during peak business hours.
• Deployment State: More aggressive shedding during a canary deployment or infrastructure change.
• Business Context: Adjusting priority classifications in real-time (e.g., during a sales event). This moves beyond static configuration, allowing the system to autonomously optimize its resilience posture in response to changing operational conditions.

Load shedding is a resilience pattern where a system under stress selectively rejects incoming requests to prevent overload and maintain service for its most critical functions. It acts as a proactive, upstream circuit breaker.

• Key Mechanism: The system implements a shedding policy that defines which requests to drop (e.g., based on priority, type, or client).
• Goal: Preserve system stability and core functionality by sacrificing non-essential work, preventing a cascading failure that could result from resource exhaustion (CPU, memory, I/O).
• Analogy: Similar to an electrical grid shedding non-critical loads to prevent a total blackout.

In multi-agent systems, load shedding is critical for managing concurrent tool calls, API dependencies, and inter-agent communication that can create bottlenecks.

• Agent-Level Shedding: An individual agent may shed lower-priority sub-tasks or defer non-urgent reasoning steps when its internal resource monitors indicate high load.
• Orchestrator-Level Shedding: The system's orchestrator or dispatcher can reject new agent-invocation requests or pause low-priority agent workflows.
• Dependency-Aware Shedding: Shedding decisions consider the health of downstream services (APIs, vector databases). If a critical dependency is failing, the system may shed requests that rely on it to avoid queueing and timeouts.

The logic determining what to shed is defined by a policy. Common strategies include:

• Priority-Based: Requests are tagged with a priority level (e.g., critical, high, low). Low-priority requests are shed first.
• Type-Based: Non-essential operation types (e.g., a 'generate summary' request) are shed before core operations (e.g., a 'process transaction' request).
• Client-Based: Traffic from certain non-essential client applications or user tiers is shed.
• Random Drop: A simple, stateless method where a percentage of incoming requests are randomly dropped under load.
• Queue Management: Shedding requests from the head (oldest) or tail (newest) of the work queue, each with different latency/ fairness implications.

Load shedding is often confused with similar resilience patterns. Key distinctions are:

• vs. Circuit Breaker: A circuit breaker stops all traffic to a failing dependency after an error threshold is crossed. Load shedding proactively drops some traffic before total failure, based on load metrics.
• vs. Rate Limiting: Rate limiting caps the number of requests per time window for fairness or cost control. Load shedding is a reactive survival mechanism triggered by system overload, not a constant cap.
• vs. Bulkhead: Bulkheads isolate failures to a pool of resources. Load shedding manages the inflow of work to prevent those pools from being overwhelmed in the first place.
• vs. Graceful Degradation: Degradation reduces feature quality. Shedding reduces quantity of work by outright rejecting requests.

Effective load shedding requires precise monitoring to decide when to activate.

• Primary Triggers:
  • Resource Utilization: CPU > 90%, memory pressure, high I/O wait times.
  • Queue Depth: The backlog of pending requests exceeds a threshold.
  • Latency Percentiles: The 95th or 99th percentile response time degrades beyond a Service Level Objective (SLO).
  • Downstream Health: Degradation or failure of a critical dependent service.
• Implementation: Triggers are often based on metrics from application performance monitoring (APM) tools or custom health check endpoints. The system must react quickly, often using a static threshold or a simple adaptive algorithm.

Consider an AI API Gateway handling requests for multiple models and agents.

Scenario: A surge in traffic hits the text-generation endpoint, causing high latency.

Load Shedding Response:

1. The gateway's monitoring detects latency exceeding the 500ms SLO for the text-generation endpoint.
2. The shedding policy activates: all new requests to the /v1/chat/completions endpoint with a priority: low header are immediately rejected with a HTTP 503 Service Unavailable status.
3. Concurrently, high-priority requests from paid enterprise clients and all traffic to the critical transaction-classification agent continue to be processed.
4. Once metrics return to normal (e.g., latency < 300ms for 30 seconds), the shedding policy is lifted, and all request types are accepted again.

This prevents the gateway from becoming unresponsive to all clients.

A software design pattern that detects failures and prevents an application from repeatedly attempting an operation that is likely to fail. It functions like an electrical circuit breaker with three states:

• Closed: Requests flow normally.
• Open: Requests fail immediately without attempting the operation.
• Half-Open: A limited number of test requests are allowed to probe for recovery. Its primary goal is to stop cascading failures and allow a failing downstream service time to recover, complementing load shedding by stopping traffic at the source.

A system design principle where functionality is reduced in a controlled manner when under failure or resource constraints. Unlike binary failure, the system maintains core operations while non-essential features are disabled. For example:

• A video streaming service reduces resolution but does not stop playback.
• An e-commerce site disables product recommendations but keeps the checkout flow active. This is the user-facing outcome that load shedding aims to achieve: preserving critical user journeys by proactively shedding non-critical load.

A flow control mechanism where a component that is overwhelmed signals upstream producers to slow down or stop sending data. It is a reactive, propagating signal rather than a point-in-time decision. Key implementations include:

• Reactive Streams: A standard for asynchronous stream processing with non-blocking backpressure.
• TCP Windowing: The receiver advertises its available buffer size to the sender. While load shedding drops requests at the ingress, backpressure manages the data flow between internal components to prevent buffer overflows and memory exhaustion.

A resilience pattern that isolates elements of an application into pools so that a failure in one pool does not cascade to others. It is inspired by ship bulkheads that contain flooding. Key implementations:

• Thread Pool Isolation: Dedicated thread pools for different service calls (e.g., payment vs. recommendation service).
• Resource Partitioning: Separate connection pools, memory allocations, or even process boundaries. This pattern complements load shedding by ensuring that when load is shed from one failing component, the resources and failure are contained, protecting other healthy parts of the system.

A technique to control the rate of requests sent or received by a network, API, or service. It is typically used for:

• Fair Usage: Preventing a single user from monopolizing resources.
• Cost Control: Managing API call costs against a third-party service.
• Security: Mitigating brute-force attacks. Key difference from Load Shedding: Rate limiting is a constant, policy-driven cap applied regardless of system health. Load shedding is a dynamic, health-reactive measure that activates only under excessive load, often prioritizing request types rather than just capping volume.

Mechanisms for continuously assessing the operational status of system components to inform resilience actions.

• Health Check: A periodic diagnostic request (e.g., /health) to verify liveness and readiness.
• Outlier Detection: A dynamic mechanism (common in service meshes) that identifies and ejects unhealthy hosts from a load balancing pool based on metrics like consecutive failures. These are the sensing systems that provide the real-time data (failure rates, latency) necessary to make intelligent load shedding decisions, determining when to shed and which backend instances are failing.