Load Shedding

The foundational mechanism for load shedding. Incoming requests are assigned a priority score based on attributes like user tier, request type, or associated Service Level Agreement (SLA). A system under load uses this score to make real-time admission decisions.

• High-Priority Requests: Always admitted for processing.
• Medium-Priority Requests: Admitted if capacity exists; may be queued.
• Low-Priority Requests: Deliberately rejected (shed) first when the system approaches overload.

This ensures critical business functions, like a premium customer's transaction, are guaranteed resources, while non-essential batch processing can be deferred.

A critical companion policy to server-side shedding. When a request is rejected (HTTP 429 or 503), the client application should not retry immediately, as this worsens the overload. Instead, it implements exponential backoff.

• The client waits for a short, random interval before retrying (e.g., 1 second).
• If rejected again, the wait time doubles for each subsequent attempt (e.g., 2s, 4s, 8s).
• This graceful degradation spreads retry attempts over time, allowing the server to recover and preventing a retry storm that can cause a full outage.

Manages requests that have been admitted but are waiting in a request queue. Without management, old requests can consume resources needed for new, higher-priority ones.

• Maximum Queue Depth: A hard limit on queue length; new requests are shed if the queue is full.
• Request Timeouts: Each request has a client-specified or system-default deadline.
• Deadline-Aware Eviction: The system can proactively shed queued requests that are predicted to miss their deadline, freeing capacity for requests that can still succeed on time. This optimizes the success rate for admitted work.

Load shedding is activated by monitoring system health metrics. Proactive triggers prevent reactive, panicked shedding after the system is already failing.

Key triggers include:

• Resource Utilization: GPU memory > 90%, CPU load > 80%.
• Latency Degradation: P95 response time exceeding SLA threshold.
• Error Rate Increase: A rising percentage of failed requests.
• Queue Growth Rate: The request queue is filling faster than it's being drained.

When a trigger threshold is breached, the shedding policy activates at a predefined severity level, scaling up the percentage of low-priority requests shed.

This highlights the strategic advantage of load shedding. Global throttling (like a rate limit) indiscriminately rejects a percentage of all traffic, harming high and low-priority users alike.

Differentiated Load Shedding is a targeted approach:

• User/Endpoint Segmentation: API endpoints for real-time chat are protected, while batch summarization endpoints are shed.
• Tenant Isolation: Traffic from a single malfunctioning or abusive tenant can be shed without impacting others.
• Cost-Aware Shedding: Requests that consume disproportionate resources (e.g., very long context windows) may be shed first.

This maximizes business value preserved during an overload incident.

Load shedding is part of a broader cost and performance management loop. It works in concert with autoscaling.

• Shedding as a Buffer: Shedding handles sudden, unpredictable traffic spikes that are too fast for autoscaling to react to (which takes minutes to spin up new instances).
• Informing Scale-Up Decisions: A high rate of shedding can be a signal to the autoscaler to increase the instance count more aggressively.
• Preventing Cost Spikes: By rejecting non-critical work, shedding prevents the system from scaling out to extremely expensive levels to handle unsustainable load, directly controlling burst capacity costs. It defines the performance-cost tradeoff explicitly.

Quality of Service (QoS) is a set of policies that prioritize certain inference requests or user groups to guarantee minimum performance levels. It is the proactive counterpart to reactive load shedding.

• Mechanisms include priority queues, reserved capacity for VIP users, and network bandwidth allocation.
• Trade-off: Implementing strict QoS often reduces overall system throughput and increases cost per request for non-priority traffic.
• Relationship to Load Shedding: QoS defines which requests get served during normal operation; load shedding defines which requests get dropped during overload to protect the QoS guarantees for high-priority traffic.

SLO Compliance measures the percentage of time an inference service meets its predefined performance targets, such as P99 latency or throughput. It is the primary business metric load shedding is designed to protect.

• A common SLO is "95% of requests complete within 200ms."
• Load shedding directly impacts SLOs: By shedding low-priority traffic, the system reduces queueing delay for remaining requests, increasing the chance they meet their latency SLO.
• Cost of Non-Compliance: Violating SLOs can trigger financial penalties in customer contracts and damage service reputation.

Request Queuing is the mechanism that temporarily holds incoming inference requests in a buffer (queue) when all model instances are busy. It is the precursor state to load shedding.

• Purpose: Queues smooth traffic bursts and enable efficient continuous batching.
• Queue Management: Algorithms like FIFO (First-In, First-Out) or priority-based scheduling determine the order of execution.
• Overflow: When a queue exceeds its maximum configured length, new incoming requests must be either shed (load shedding) or routed to a failover system.

Burst Capacity is the temporary, maximum additional throughput an inference system can handle beyond its sustained operational baseline. It defines the safety margin before load shedding is required.

• Enabled by: Spare (overprovisioned) resources, rapid autoscaling, or borrowing capacity from other services.
• Engineering Trade-off: Higher burst capacity increases resilience to usage spikes but also raises baseline infrastructure costs due to idle resources.
• Load Shedding Trigger: When incoming traffic exceeds the system's sustained capacity plus its available burst capacity, load shedding is activated as a last line of defense.

Autoscaling is an automated cloud infrastructure technique that dynamically adjusts the number of active compute instances (e.g., GPU servers) in response to changes in inference traffic. It is a primary method for preventing the need for load shedding.

• Reactive Scaling: Adds instances after a traffic increase is detected, but suffers from cold start latency.
• Predictive Scaling: Uses workload prediction to provision resources ahead of anticipated demand.
• Limitation: Autoscaling has physical and financial limits (e.g., max instances, budget caps). When scaling cannot keep pace with a sudden spike, load shedding acts as a circuit breaker.

Cost-Per-Token is the fundamental financial metric for inference, calculating the expense to generate a single token. Load shedding is a cost-control mechanism that directly impacts this metric during overload.

• Calculation: (Instance Cost per Hour) / (Tokens Generated per Hour).
• Under Load: As queues grow, latency increases but tokens/hour may stay flat, effectively raising the cost-per-token for all users.
• Load Shedding Effect: By rejecting some requests, the system maintains high tokens/hour throughput for accepted requests, protecting the cost-per-token metric for high-priority traffic and preventing runaway costs from inefficient, overloaded processing.