Thundering Herd Problem

The anti-pattern is initiated when a hot cache key—holding a frequently requested AI inference, embedding, or tokenization result—reaches its Time-To-Live (TTL) and expires. Instead of a single process regenerating the value, hundreds or thousands of concurrent client requests simultaneously detect the cache miss. Each request independently initiates a full recomputation, such as a forward pass through a large language model. This transforms a single cache refresh into a self-inflicted Denial-of-Service (DoS) attack on the backend model serving infrastructure.

The initial overload causes request timeouts, which triggers client-side exponential backoff retry logic. However, without jitter and coordination, these retries synchronize into distinct, phased waves of traffic. Each subsequent retry wave compounds the load on an already degraded system. In AI inference clusters, this manifests as a cascading failure: saturated GPU memory queues lead to out-of-memory (OOM) kills, forcing pod restarts that further reduce cluster capacity and increase the load on surviving replicas.

The thundering herd consumes all available connection pool slots and request queue capacity. This creates a priority inversion where lightweight health checks and critical system control plane operations are starved of resources by the heavy, redundant inference requests. Kubernetes liveness probes fail due to the overload, causing the orchestrator to incorrectly mark overloaded pods as unhealthy and restart them. This control-plane starvation loop prevents the system from self-healing, as new pods are immediately crushed by the unrelenting herd of pending requests.

A standard mitigation is probabilistic early recomputation. When a cached AI response approaches its expiry, the system calculates a random probability of proactively refreshing it. As the expiry deadline nears, this probability increases. For example, a process might roll a die at 80% of TTL; if it wins, it refreshes the cache while the stale value is still served. This ensures that only one or a few processes ever attempt the expensive recomputation, completely eliminating the synchronized stampede by distributing the refresh decision across time.

For systems that cannot use probabilistic expiration, request coalescing (also known as request collapsing) is a critical defense. A dedicated middleware layer or a single-flight lock detects multiple identical requests for the same cacheable resource. It executes only one backend call to the AI model and holds all other concurrent requests in a waiting state. Once the single computation completes, the result is broadcast to all waiting clients simultaneously. This reduces N identical LLM calls to exactly 1, preserving GPU compute for unique, non-cacheable work.

When coalescing fails, the last line of defense is load shedding. A circuit breaker monitors the error rate and latency of downstream AI services. If the failure threshold is breached, the breaker trips to an open state, immediately failing all requests without forwarding them to the struggling backend. This fast-fail mechanism preserves server resources by rejecting work at the edge. Combined with priority queues, the system can shed low-priority batch inference jobs while continuing to serve critical real-time requests, preventing total system collapse.

A specific instance of the thundering herd problem where multiple concurrent requests attempt to regenerate a cached value simultaneously after it expires. When a hot cache key expires, all waiting requests trigger identical, redundant computations to repopulate it, overwhelming backend resources.

• Cause: High-traffic keys expiring without a refresh-ahead strategy
• Impact: CPU spikes, database connection exhaustion, cascading failures
• Common in: CDN edge nodes, database query caches, AI model inference caches

A retry strategy where clients progressively increase the wait time between consecutive retry attempts after a failure. In the context of the thundering herd, adding jitter (randomized delay) to backoff prevents synchronized retry storms.

• Formula: delay = min(cap, base × 2^attempt)
• Jitter: Randomizes timing to desynchronize competing clients
• Use case: API clients retrying after a 503 Service Unavailable response

A cache management technique where the system preemptively recomputes a cached value before its actual expiration. When a request arrives within a defined window before expiry, the system has a probability of triggering an asynchronous refresh.

• Algorithm: XFetch computes optimal refresh probability based on request rate
• Benefit: Prevents all requests from encountering a cold cache simultaneously
• Trade-off: May perform unnecessary recomputations during low-traffic periods

A concurrency pattern where multiple identical requests are merged into a single backend operation. The first request initiates the work, and subsequent requests wait on the same promise or future rather than spawning duplicate computations.

• Implementation: Single-flight pattern using a deduplication map keyed by request signature
• Result: N concurrent cache misses produce exactly 1 backend call
• Frameworks: Go's singleflight, Rust's deduplicate crate

A resilience pattern that detects downstream failures and temporarily stops sending requests to a failing service. When the thundering herd overwhelms a backend, the circuit breaker opens, failing fast and allowing the system to shed load rather than compounding the problem.

• States: Closed (normal), Open (failing fast), Half-Open (testing recovery)
• Metrics tracked: Failure rate, latency, request volume
• Prevents: Resource exhaustion from retry amplification

A concurrency control approach that avoids mutual exclusion entirely, using atomic operations to manage shared state. In high-throughput AI serving systems, lock-free data structures prevent the contention and priority inversion that can trigger thundering herd cascades.

• Primitives: Compare-and-swap (CAS), fetch-and-add
• Advantage: Guarantees system-wide progress without deadlock risk
• Relevance: Critical for real-time inference where mutex contention causes tail latency spikes