Cache Stampede

Instead of using a fixed Time-To-Live (TTL) , the cache layer randomly expires entries within a window before the nominal deadline. When a request encounters an entry nearing its logical expiry, it proactively recomputes the value while still serving the stale data to other requestors.

• Prevents synchronized mass expiry of popular keys
• Distributes recomputation load across a time window
• Works best when combined with exponential backoff on misses

A circuit breaker monitors the health of the origin backend. When failure rates or latency exceed a threshold, the breaker trips and the cache layer serves stale data beyond its nominal TTL rather than forwarding requests to a failing origin.

• Implements graceful degradation under load
• Requires application tolerance for eventually consistent data
• Automatically resets when the origin recovers

The cache proxy or middleware collapses multiple identical in-flight requests into a single origin query. When the response returns, it is fanned out to all waiting clients. This is a fundamental pattern in Nginx, Varnish, and Envoy.

• Reduces origin load by orders of magnitude during spikes
• Transparent to client applications
• Often combined with token bucket rate limiting for defense in depth

A background worker monitors hot keys and proactively refreshes them before expiry. When a key's remaining TTL drops below a threshold, the worker asynchronously recomputes the value and updates the cache, ensuring the entry never actually expires under normal conditions.

• Eliminates cache misses for predictable, high-traffic keys
• Requires knowledge of access patterns or manual key registration
• Pairs well with cache prefetch strategies

A token bucket algorithm placed in front of the origin enforces a strict maximum request rate. Tokens are generated at a fixed rate and consumed by each forwarded request. When the bucket is empty, excess requests are immediately rejected with a 429 Too Many Requests status.

• Protects the origin from any surge, stampede or otherwise
• Provides backpressure that propagates to clients
• Can be implemented at the API gateway or service mesh layer

An optimization where the cache recomputes an expired entry early with a certain probability before its TTL expires, rather than waiting for expiration to trigger a synchronous refresh.

• Uses exponential decay to increase refresh probability as expiry approaches
• Eliminates the thundering herd at the expiration boundary
• Commonly implemented in XFetch and PERC algorithms
• Trades a small amount of redundant computation for dramatic latency reduction

A technique where multiple concurrent requests for the same cache key are collapsed into a single backend fetch, with all waiting clients served the same result.

• Implements a promise deduplication pattern
• Uses a lock or single-flight mechanism per key
• Dramatically reduces origin load during cold starts
• Critical for high-traffic sovereign inference endpoints

A retry strategy where clients increase the delay between retry attempts exponentially, adding random jitter to desynchronize competing requests.

• Formula: delay = min(cap, base × 2^attempt) + random_jitter
• Prevents self-synchronization of retrying clients
• AWS, Google Cloud, and Stripe all mandate jitter in their SDKs
• Essential for distributed cache clients recovering from cold starts

An extension of the cache-aside pattern where the application acquires a distributed lock before populating the cache on a miss, ensuring only one caller regenerates the value.

• Uses Redis Redlock or etcd for distributed coordination
• Other callers either wait or serve stale data
• Prevents the simultaneous recomputation that defines a stampede
• Requires careful lock timeout tuning to avoid deadlocks