Load Shedding

Load shedding operates on a spectrum from proactive to reactive strategies.

• Proactive (Predictive) Shedding: The system uses metrics like queue depth, latency percentiles (e.g., p99), or upstream error rates to predict impending overload and begins shedding traffic before critical thresholds are breached. This is akin to a circuit breaker moving to an open state based on trip conditions.
• Reactive Shedding: The system sheds traffic only after a resource limit is hit, such as CPU utilization exceeding 95% or memory pressure triggering an out-of-memory (OOM) killer. This is a last-ditch effort to avoid a crash.

Effective systems implement both, using predictive models for graceful degradation and reactive measures as a final safety net.

The decision of what to shed is fundamental. Criteria must be deterministic and quickly evaluable.

• Request Priority: Incoming requests are tagged with metadata (e.g., HTTP header X-Priority: {CRITICAL, HIGH, LOW}) or classified by endpoint (e.g., /api/health is critical, /api/report/generate is low).
• User/Client Tier: Traffic from premium or internal users is preserved over free-tier or external users.
• Request Cost: Computationally expensive requests (e.g., complex LLM inferences, large file processing) are shed before simple health checks or cache lookups.
• Request Freshness: Older queued requests may be dropped first if they are likely to have timed out client-side.

The classification logic must be a fast, in-memory operation to avoid adding significant overhead during the load event.

Load shedding is implemented at various layers in a system architecture.

• Edge/Load Balancer Layer: Global load balancers (e.g., cloud provider LBs) can shed traffic based on simple health signals before it reaches application servers. This is often combined with health check endpoints.
• API Gateway/Service Mesh Layer: Sidecar proxies in a service mesh (e.g., Istio, Linkerd) or API gateways can enforce fine-grained shedding policies per service, using patterns like the bulkhead pattern to isolate failures.
• Application Layer: The service itself implements shedding logic, often using a token bucket or leaky bucket algorithm to control admission. This allows for the most nuanced classification based on business logic.
• Queue-Based Systems: For event-driven architectures, load shedding involves configuring dead letter queues (DLQs) for messages that cannot be processed after retries, and setting appropriate queue depth alarms to trigger scaling or alerting.

How a system communicates shedding decisions is crucial for overall stability.

• HTTP Status Codes: The standard response for a shed request is HTTP 503 Service Unavailable, often with a Retry-After header indicating when to retry. This is preferable to a timeout or connection drop, as it is an explicit signal.
• Backpressure Propagation: In streaming or reactive systems, shedding is a form of backpressure. The overwhelmed node signals upstream to slow down, propagating the shedding decision back through the data flow graph.
• Client Retry Logic: Clients must implement intelligent retry strategies, such as exponential backoff with jitter, to avoid a retry storm where all clients simultaneously retry, worsening the overload. Idempotent operations are essential for safe retries.
• Graceful Degradation: User interfaces should be designed to handle 503 responses gracefully, showing appropriate messaging and disabling non-critical features.

Load shedding events must be intensely monitored to tune policies and understand system behavior.

• Key Metrics:
  • Shedding rate (requests rejected/sec).
  • Error budget consumption (SLO violation).
  • Resource utilization at the moment of shedding (CPU, memory, queue depth).
  • Latency percentiles for accepted vs. shed request paths.
• Alerting: Alerts should fire not just when shedding occurs, but when it occurs at a sustained rate, indicating a chronic capacity issue rather than a transient spike.
• Distributed Tracing: Traces must capture the shedding decision point, annotating spans with tags like load_shed: true and shedding_priority: low. This allows for post-incident analysis of what user journeys were impacted.
• Chaos Engineering: Fault injection tests should validate that shedding policies activate correctly under controlled failure conditions and that the system stabilizes as expected.

Load shedding is one tool in a broader resilience toolkit and must be coordinated with other patterns.

• Circuit Breaker: A circuit breaker prevents calls to a failing downstream service. Load shedding prevents a service from being overwhelmed by upstream calls. They are complementary: a circuit breaker protects you from them; load shedding protects you from everyone else.
• Rate Limiting: Rate limiting is proactive and static, defining a hard cap for a user or client. Load shedding is dynamic and system-wide, triggered by overall health. Rate limiting is often a precursor; if global limits are exceeded, shedding begins.
• Autoscaling: Shedding is a immediate, stateless reaction. Autoscaling is a slower, stateful response to add capacity. Shedding "holds the line" while scaling catches up. If shedding is constantly active, it indicates an autoscaling policy or capacity limit needs adjustment.
• Graceful Degradation: Load shedding is a primary mechanism to enable graceful degradation. By shedding non-critical features, the system ensures critical ones remain available, maintaining a useful, albeit reduced, service level.

For real-time AI services like chatbots, fraud detection, or autonomous systems, predictable latency is a non-negotiable SLA. Under load, shedding non-critical requests (e.g., lower-priority users, batch inference jobs) ensures that high-priority, latency-sensitive requests are processed within their required time window. This prevents a latency death spiral where queued requests cause timeouts for all users.

• Example: A video streaming service sheds requests for generating personalized thumbnails to guarantee sub-100ms latency for its real-time content recommendation engine.

GPU/TPU instances are expensive and finite. A sudden traffic surge can exhaust video memory (VRAM) or cause out-of-memory (OOM) errors, crashing the entire model server. Load shedding acts as a pressure relief valve by rejecting requests before hardware limits are breached.

• Mechanism: A serving system monitors GPU memory utilization and queue depth. When thresholds are crossed, it begins shedding requests based on a priority score or user tier, preserving capacity for critical inference workloads and preventing a costly, cascading service outage.

In multi-agent systems or complex agentic workflows, an agent may call multiple tools (APIs, databases, other models). If a downstream dependency is slow or failing, the agent's execution can hang. Load shedding at the tool-calling layer involves dropping non-essential tool calls or switching to fallback tools to complete the core objective.

• Example: A customer service agent prioritizing a 'process refund' tool call over a 'fetch user history' call when the database is under heavy load, ensuring the primary transactional action succeeds.

AI inference costs scale directly with usage. An unexpected viral event or a misconfigured client can trigger a cost avalanche. Proactive load shedding based on cost-per-request budgets or rate limits prevents runaway expenses. This is often implemented alongside autoscaling, but shedding is faster and more cost-effective than spinning up new, expensive accelerator instances.

• Implementation: A gateway tracks cost per model invocation and total daily spend. When projected costs exceed budget, it begins to shed requests for the most expensive models first, or for non-contractual users.

In continuous learning or online training systems, data ingestion and preprocessing pipelines must keep pace with incoming data. If feature computation or data validation becomes a bottleneck, backpressure can cause data loss. Load shedding in this context means sampling or discarding low-value training data to preserve pipeline throughput for high-fidelity data, ensuring model updates remain timely and based on the most important signals.

• Use Case: An anomaly detection system for network security temporarily stops ingesting low-severity log data during a DDoS attack to ensure real-time processing of high-severity threat signals.

Complex multi-modal AI systems (e.g., combining vision, language, and audio) have multiple, potentially resource-intensive, model pathways. Under load, the system can shed entire modalities or downgrade model resolution to maintain a baseline service level.

• Scenario: A video analysis service under load might:
  • Shed the audio transcription model and process only visual frames.
  • Switch from a large Vision Transformer (ViT) to a smaller, faster convolutional model.
  • Return a text-only summary instead of a full multi-modal report. This controlled degradation is preferable to a complete system failure.

A design pattern that prevents a software component from repeatedly attempting an operation that is likely to fail. It functions like an electrical circuit breaker:

• Closed State: Requests flow normally.
• Open State: Requests fail immediately without attempting the operation, after failure thresholds are met.
• Half-Open State: Allows a limited number of test requests to see if the underlying issue is resolved. This pattern stops cascading failures and allows downstream systems like load balancers to redirect traffic, working in concert with load shedding to maintain system stability.

A design pattern that isolates elements of an application into independent resource pools (bulkheads). If one component fails or is overloaded, the failure is contained, preventing it from cascading and exhausting resources for the entire system. This is a proactive architectural complement to load shedding.

• Implementation: Uses separate thread pools, connection pools, or even microservices for different client classes or operations.
• Benefit: Allows non-critical functions to fail gracefully while preserving capacity for critical operations, making load shedding decisions more granular and effective.

A flow control mechanism where a fast data source is signaled to slow down or stop sending data when a downstream component is unable to keep up. This prevents buffer overflow, system collapse, and uncontrolled resource exhaustion.

• Reactive Streams: Frameworks like Project Reactor or Akka Streams implement formal backpressure protocols.
• Relation to Load Shedding: While load shedding actively drops incoming requests at the system boundary, backpressure propagates saturation signals backward through the data pipeline. They are often used together: backpressure manages internal flow, while load shedding protects the system's front door.

A system design principle where functionality is reduced in a controlled, deliberate manner when a component fails or resources are constrained. The goal is to preserve core operations and a functional user experience, even if in a limited form.

• Examples: A search engine returning results without personalized rankings; a video stream reducing resolution.
• Strategic Difference: Load shedding is a specific tactic (dropping requests) to enable graceful degradation. Degradation defines what level of service is maintained after shedding non-critical load.

A technique for controlling the rate of traffic sent or received by a network interface, application, or user. It protects services from excessive use—whether malicious (DDoS) or accidental—and ensures fair resource allocation.

• Common Algorithms: Token bucket, leaky bucket, fixed window, sliding window log.
• Proactive vs. Reactive: Rate limiting is typically a proactive, constant guardrail applied per user/API key. Load shedding is a reactive, system-wide emergency measure triggered by overall resource exhaustion (e.g., CPU, memory). They are layered defenses.

A dedicated API endpoint (e.g., /health or /ready) that returns the operational status of a service. It is used by load balancers, orchestrators (Kubernetes), and service discovery systems to determine if a service instance can receive traffic.

• Liveness Probe: Indicates the service process is running.
• Readiness Probe: Indicates the service is ready to accept requests (e.g., dependencies connected).
• Critical for Load Shedding: Orchestrators use failing health checks to stop routing traffic to unhealthy instances, which is a form of automated, infrastructure-level load shedding. This allows the system to isolate failing nodes.