Backpressure

The foundational asynchronous, non-blocking backpressure protocol defined by the Reactive Manifesto. It enables a subscriber to signal to a publisher exactly how many elements it can process via request(n) demand signals.

• Dynamic Push-Pull: The publisher never sends more data than the subscriber has requested, preventing buffer overflow.
• Bounded Queues: All internal buffers have strictly defined capacities; when full, upstream producers are throttled.
• TCP Backpressure: At the network layer, if the receiver's socket buffer fills, the TCP window size shrinks to zero, forcing the sender to pause transmission.

While often conflated, these are distinct overload strategies. Load shedding is a terminal decision—the server intentionally drops a request to protect itself. Backpressure is a cooperative negotiation—it delays the request upstream, preserving it for later processing.

• Load Shedding: Returns an immediate 503 or 429 status code. The client must retry.
• Backpressure: The connection is held open; the client waits. No work is lost.
• Combined Strategy: Systems often use backpressure as the first line of defense, resorting to load shedding only when the backpressure queue itself overflows or a deadline is breached.

gRPC leverages HTTP/2's built-in flow control to implement backpressure at the transport layer. Each stream and the entire connection have independent flow control windows.

• Per-Stream Limits: A single slow stream cannot block others multiplexed on the same TCP connection.
• WINDOW_UPDATE Frames: The receiver sends these frames to grant the sender more send window capacity.
• Blocking Stubs: In synchronous gRPC clients, a full send buffer blocks the calling thread, naturally propagating backpressure to the application logic.

Backpressure signals can be externalized to drive infrastructure scaling. The Horizontal Pod Autoscaler (HPA) can consume custom metrics derived from backpressure indicators.

• Queue Depth Metric: The number of requests waiting in the server's listen queue is exported to Prometheus.
• HPA Target: A rule like queue_depth > 5 triggers the creation of new model serving pods.
• Stabilization Window: A scale-up cooldown prevents flapping, ensuring backpressure is a sustained condition, not a transient spike, before allocating new compute resources.

Backpressure and the Circuit Breaker pattern are complementary. Backpressure is a healthy, temporary state of flow control. A circuit breaker trips when backpressure fails or the downstream system is genuinely unhealthy.

• Half-Open State: After tripping, the circuit breaker allows a limited number of probe requests. If they succeed, the circuit closes; if they fail or timeout, it remains open.
• Fallback Logic: While the circuit is open, callers execute a fallback—returning cached data, a default prediction, or a static recommendation—instead of queuing indefinitely.

Backpressure in request-serving systems mirrors the Additive Increase/Multiplicative Decrease (AIMD) algorithm used in TCP congestion control.

• Additive Increase: When no backpressure is signaled, clients slowly increase their sending rate by a fixed amount.
• Multiplicative Decrease: Upon receiving a backpressure signal (e.g., a 429 status or a full stream window), the client immediately halves its sending rate.
• Fairness: This algorithm ensures that multiple competing clients converge to a fair share of the server's capacity without centralized coordination.

A stability pattern that automatically stops sending requests to a failing model endpoint and redirects traffic to a fallback. Unlike backpressure—which signals slowdown—a circuit breaker trips open when failure thresholds are exceeded, preventing resource exhaustion. Common states include:

• Closed: Normal operation, requests flow freely
• Open: Requests are immediately rejected without attempting the call
• Half-Open: A limited number of probe requests test if the downstream service has recovered This pattern enables graceful degradation rather than complete system collapse.

A control mechanism that restricts the number of inference requests a client can make within a specific time window. While backpressure is a reactive signal from the server, rate limiting is a proactive policy enforced at the API gateway. Common algorithms include:

• Token Bucket: Allows bursts up to a defined capacity, refilling at a steady rate
• Sliding Window Log: Tracks request timestamps to enforce precise limits
• Leaky Bucket: Smooths bursty traffic into a constant outflow rate Rate limiting protects the serving infrastructure from abuse and overload before saturation occurs.

A defensive resilience strategy where a server intentionally drops a fraction of incoming requests when it detects overload. Unlike backpressure—which asks clients to slow down—load shedding makes a unilateral decision to reject work. Key design considerations:

• Priority-based shedding: Drop low-priority requests first, preserving critical traffic
• Queue-based triggers: Shed when the request queue depth exceeds a threshold
• Deadline-aware shedding: Drop requests that cannot possibly meet their SLO given current latency The goal is to prioritize successful processing of remaining traffic over total failure.

A specific, measurable target for system performance that defines the acceptable level of service reliability. Backpressure mechanisms are calibrated to activate before SLOs are violated. Common SLOs for model serving:

• P99 latency < 200ms: 99% of requests served faster than this threshold
• Availability > 99.95%: Maximum acceptable downtime per month
• Error rate < 0.1%: Proportion of failed requests allowed SLOs provide the quantitative foundation for tuning backpressure thresholds, circuit breaker trip points, and autoscaling policies.

A Kubernetes mechanism that automatically scales the number of running model replicas based on observed metrics. While backpressure signals saturation, HPA provides the remedial action by adding capacity. Scaling can be driven by:

• CPU utilization: Scale when average CPU exceeds a target percentage
• Custom metrics: Scale based on request queue depth or inference latency
• External metrics: Scale based on Pub/Sub message backlog or custom signals Effective HPA configuration works in tandem with backpressure to maintain throughput under varying load without over-provisioning.

A technique that maintains a cache of reusable, persistent network connections to backend services. Each new TCP and TLS handshake adds significant latency overhead. Connection pooling eliminates this by:

• Pre-establishing connections: Warm connections ready for immediate use
• Connection reuse: Returning connections to the pool after each request completes
• Health checking: Periodically validating idle connections before reuse In high-throughput serving, connection pooling reduces tail latency and prevents connection exhaustion, complementing backpressure by ensuring efficient resource utilization.