Load Shedding

Load shedding is a proactive control mechanism. Unlike reactive patterns that respond to failures (e.g., circuit breakers), load shedding anticipates overload by monitoring system health metrics like CPU utilization, memory pressure, queue depth, or latency percentiles. It rejects requests before resources are fully exhausted to prevent a total system collapse. This is akin to an aircraft shedding weight to avoid stalling, rather than deploying a parachute after a crash.

The strategy is defined by intelligent selectivity. Not all traffic is treated equally. Systems implement priority queues or request classifiers to identify and shed non-critical work. Common criteria include:

• Request Type: Dropping read-only or batch API calls before write operations.
• Client Tier: Prioritizing premium or internal users over free-tier or external traffic.
• Resource Cost: Shedding computationally expensive requests (e.g., complex reports) before simple lookups.
• Deadline Awareness: Rejecting requests with looser latency SLOs first. The goal is to preserve capacity for critical-path transactions that directly impact business continuity.

Load shedding enables graceful degradation. The system intentionally reduces its functional scope to maintain core availability. Instead of a generic 503 error, it should provide a clear, immediate signal to the client, such as an HTTP 429 (Too Many Requests) or 503 (Service Unavailable) status code with a Retry-After header. This allows clients and upstream systems to implement their own fallback logic, such as failing over to a secondary region, returning cached data, or informing users of a temporary limitation. The system remains partially operational, unlike in a total outage.

Effective load shedding is deeply integrated with observability tooling. It requires real-time visibility into key metrics to make accurate shedding decisions. This includes:

• Golden Signals: Traffic, errors, saturation, and latency.
• Resource Metrics: Garbage collection pressure, thread pool utilization, and database connection pools.
• Business Metrics: Transaction success rates for critical user journeys. These metrics feed into the shedding controller, which uses thresholds or adaptive algorithms to trigger the behavior. All shed requests must be logged and audited to analyze shedding effectiveness and adjust policies.

Load shedding is often conflated with throttling and rate limiting, but they serve different purposes:

• Rate Limiting: A preventative, constant-enforcement policy applied per client/API key to prevent abuse and ensure fair usage (e.g., 100 requests/minute).
• Throttling: A dynamic, reactive slowing of request processing to match system capacity (e.g., delaying queue processing).
• Load Shedding: An emergency overload response that outright rejects requests to protect the system's core function. It is the last line of defense when throttling and scaling are insufficient. Shedding is triggered by system state, not client identity.

Common technical implementations include:

• Queue Management: Using a bounded queue with a reject policy (e.g., ThreadPoolExecutor.CallerRunsPolicy or custom rejection handlers).
• Middleware/Proxy Layer: Implementing logic in an API gateway or service mesh (e.g., Envoy's Load Shedding filters) to reject requests before they reach the application.
• Adaptive Algorithms: More sophisticated systems use algorithms like TCP Vegas-inspired congestion control, adjusting shedding thresholds based on measured latency increases.
• Coordination with Autoscaling: Load shedding buys critical time for horizontal autoscaling to kick in, but should not be a substitute for adequate capacity planning.

A flow control mechanism where a system receiving data faster than it can process signals the upstream sender to slow down or stop. This prevents the receiving component from being overwhelmed, which is a precursor to needing load shedding. While load shedding drops requests, backpressure aims to regulate the flow before the queue becomes unmanageable.

• Reactive Streams is a standard implementation for backpressure in asynchronous systems.

Proactive controls that restrict request volume to prevent overload. Rate limiting caps the number of requests a client can make in a window. Throttling deliberately slows request processing. These are applied before a system reaches critical load, whereas load shedding is a last-resort reaction when those controls are insufficient or an unexpected surge occurs. They work together in a defense-in-depth strategy.

A design pattern that isolates application components into independent resource pools (bulkheads), similar to a ship's compartments. If one component fails or is overloaded, its pool is exhausted, but the failure is contained and does not drain resources from other components. This limits the blast radius and makes load shedding decisions more granular—you can shed traffic to a failing bulkhead while other services remain fully operational.

The design principle of maintaining partial or reduced functionality when components fail. Load shedding is a key tactic to enable graceful degradation. By shedding non-critical features (e.g., disabling profile picture uploads), the core service (e.g., login and transactions) remains available. The goal is a continuously useful, if imperfect, user experience instead of a complete outage.

A holding queue for messages or requests that cannot be processed successfully after multiple retry attempts. In the context of load shedding, rejected requests can be directed to a DLQ instead of being silently dropped. This allows for:

• Auditing and analysis of shed load.
• Manual or automated reprocessing once system health is restored.
• Maintaining a record for compliance or billing purposes.