Cache Stampede

Instead of a fixed TTL, each request computes a random probability of refreshing the cache as the expiry time approaches. This spreads out the refresh load over a time window.

• Mechanism: At read time, if current_time > TTL - (TTL * beta * random(0,1)), trigger a background recomputation.
• Key Parameter: beta controls the width of the recomputation window.
• Advantage: No external coordination service required; implemented entirely in the cache client.

The first request that detects a cache miss acquires a distributed lock before querying the backend. All other concurrent requests wait on the lock and then read the freshly populated value.

• Implementation: Typically uses Redis (SETNX), etcd, or ZooKeeper.
• Critical Detail: Lock must have a TTL to prevent deadlock if the populating process crashes.
• Trade-off: Adds latency for waiting requests but completely eliminates duplicate backend work.

Serve the expired (stale) cache entry immediately to all requestors while asynchronously triggering a single background refresh. Users never experience the cache miss latency.

• HTTP Context: Standardized via the Cache-Control: stale-while-revalidate=30 directive.
• Application Context: Cache libraries return the expired object and spawn a single thread to update it.
• Key Benefit: Eliminates latency spikes for end-users during cache expiry events.

Proactively pre-load the cache with anticipated hot keys before they are requested, often triggered by deployment or known traffic patterns.

• Process: A script or CI/CD step iterates through a manifest of critical keys and populates the cache immediately after a flush or deploy.
• Use Case: Essential for Time-To-First-Token (TTFT) optimization in RAG systems where embedding caches must be hot.
• Risk: Warming incorrect keys wastes memory and can evict genuinely useful data.

Identical concurrent requests for the same key are collapsed into a single backend call. The result is then fanned out to all waiting callers.

• Implementation: A request map tracks in-flight fetches. If a fetch for key K is already in progress, subsequent requests subscribe to the same future/promise.
• Contrast with Locking: No distributed lock infrastructure needed; works within a single service instance.
• Library Support: Built into many HTTP client libraries and frameworks like Django's cache framework.

Maintain two expiry timestamps: a soft TTL for triggering background refresh and a hard TTL for actual eviction. The soft TTL is set shorter to create a refresh buffer.

• Example: Soft TTL = 50s, Hard TTL = 60s. At 50s, a single thread refreshes the value while others continue reading the old value until 60s.
• Failure Mode: If the background refresh fails, the hard TTL eventually evicts the key, causing a miss.
• Mitigation: Combine with stale-while-revalidate to serve the hard-expired value as a last resort.

The broader concurrency failure mode of which a cache stampede is a specific instance. A thundering herd occurs when a large number of processes or threads, waiting for a single resource, are woken simultaneously and overwhelm the system. In the context of a cache stampede, the 'herd' is the flood of requests for the expired key. Mitigation involves request coalescing or jitter to desynchronize competing operations.

A proactive cache stampede prevention strategy where each read of a cached item has a small, calculated probability of triggering an early, asynchronous refresh before the key's official TTL expires. The probability increases as the expiry time approaches. This ensures that only one or a few requests perform the expensive recomputation, rather than a flood at the hard deadline. This technique is sometimes called XFetch.

A reactive mitigation technique where a barrier or lock is placed in front of the cache miss path. When the first request for an expired key arrives, it acquires a lock and begins recomputing the value. All subsequent concurrent requests for the same key check for the lock and, finding it, wait on the result of the first computation rather than starting their own. This collapses N identical backend queries into a single one.

A simple but critical technique for desynchronizing events. In caching, jitter is a small, random delay added to a key's Time-To-Live (TTL). Instead of setting a TTL to exactly 60 seconds, a system might set it to a random value between 55 and 65 seconds. This prevents multiple keys that were cached simultaneously from expiring at the exact same moment, naturally distributing the refresh load over a window of time.