Load Shedding

Load shedding is a proactive defense, not a reactive failure. It is triggered by predictive metrics (e.g., queue length, estimated processing time) before the system becomes unstable. This prevents cascading failures where latency spikes for all users, and ensures the system remains within its Service-Level Objective (SLO) for accepted requests. It's analogous to a circuit breaker in distributed systems.

Shedding decisions are governed by a defined admission control policy. Common policies include:

• Queue Length Threshold: Reject new requests if the pending queue exceeds N items.
• Latency Prediction: Reject requests whose estimated completion time would violate the SLA.
• Resource-Based: Reject based on GPU memory pressure or high utilization.
• Priority-Based: Shed lower-priority requests (e.g., batch jobs) before high-priority ones (e.g., interactive chat). Policies are configurable and central to system behavior under load.

The primary goal is to bound tail latency (p95, p99). By shedding excess load, the system avoids queueing delays that disproportionately affect all requests. This ensures predictable performance for the accepted workload. For example, shedding 10% of requests might keep p99 latency at 200ms for the remaining 90%, whereas accepting all requests could cause p99 latency to balloon to several seconds.

In systems using continuous batching, load shedding works in tandem with the scheduler. The request queue is monitored. If the queue grows faster than the iteration-level scheduler can process requests, shedding is activated. This prevents head-of-line blocking where a few long sequences stall the entire batch, and ensures the GPU utilization remains high for the processed requests.

A rejected request must be communicated clearly to the client. Standard practice is to return an HTTP 429 Too Many Requests status code, often with a Retry-After header. This allows client applications to implement graceful fallbacks, such as exponential backoff, failing over to a different endpoint, or presenting a user-friendly message. Proper signaling is essential for robust system design.

• vs. Throttling: Throttling limits request rate over time for all clients. Load shedding is an immediate, selective rejection under acute overload.
• vs. Autoscaling: Autoscaling adds resources (e.g., more GPU instances) to handle increased load, but has a long lead time (minutes). Load shedding is an instantaneous response to traffic spikes that outpace scaling. They are complementary strategies.

Request admission control is the policy layer that determines whether an incoming inference query is accepted into the system or immediately rejected. It acts as the first line of defense before load shedding. Admission decisions are based on:

• System load metrics (GPU utilization, queue depth, memory pressure).
• Request characteristics (estimated compute cost, sequence length).
• Service-Level Agreements (SLAs) (priority tiers, latency guarantees).

While load shedding actively drops already-queued requests to relieve pressure, admission control proactively prevents overload by rejecting requests at the ingress point.

Tail latency refers to the high-percentile response times (e.g., p95, p99, p99.9) of inference requests. It is the primary user-facing metric that load shedding aims to protect. In a system under load:

• Without shedding, all requests slow down, drastically inflating the tail as queues grow unbounded.
• Load shedding sacrifices a small percentage of requests (by rejecting them quickly with a clear error) to preserve predictable, low latency for the vast majority that are accepted.

This trade-off is fundamental: a controlled, small increase in error rate is preferable to an uncontrolled, large increase in latency for all users.

Backpressure is a flow-control mechanism where a downstream component (e.g., an overloaded model worker) signals upstream components (e.g., a load balancer or queue manager) to slow down or stop sending new work. It is a cooperative signal, whereas load shedding is a unilateral action.

In an inference pipeline:

1. A GPU worker nearing capacity sets a backpressure signal.
2. The scheduler reduces the rate of dispatching new batches.
3. If queues continue to grow despite backpressure, the system escalates to active load shedding to prevent collapse.

Backpressure is preventative; load shedding is reactive and more drastic.

Head-of-line (HOL) blocking is a performance problem where a single slow request within a batch prevents all other requests in that batch from completing. This is a key scenario that can trigger load shedding.

Causes include:

• A request with an extremely long output sequence.
• A computationally expensive request (e.g., with a large context window).
• A stalled request due to a downstream dependency.

Continuous batching mitigates HOL blocking by allowing finished sequences to exit the batch. However, under extreme load, a scheduler may employ load shedding to proactively identify and drop requests likely to cause severe HOL blocking, based on heuristics like requested output length.

A Service-Level Objective (SLO) is a target value for a specific service-level indicator, such as latency or throughput. Load shedding policies are directly configured to defend SLOs.

For example, an SLO might state: "99% of inference requests shall complete within 500ms."

• The system monitors real-time latency percentiles.
• When the p99 latency approaches 500ms due to high load, the load shedding policy activates.
• It begins rejecting low-priority or costly requests to bring the measured latency back within the SLO.

SLOs provide the quantitative basis for deciding when to shed load and how much load to shed.

An orchestrator in inference serving is the software component that manages the lifecycle of model instances, distributes requests across a cluster of workers, and collects system health metrics. It is the central brain that often implements load shedding logic.

The orchestrator's role in load shedding includes:

• Monitoring global metrics (aggregate queue times, error rates, hardware utilization).
• Policy Enforcement applying shedding rules based on cluster-wide state, not just a single server.
• Graceful Degradation directing traffic away from failing nodes and shedding load at the cluster ingress.

Tools like Kubernetes with custom schedulers or specialized inference orchestration platforms (e.g., Ray Serve, KServe) provide the scaffolding on which load shedding policies are built.