Cache Invalidation

Explicit invalidation involves programmatically removing or marking cache entries as invalid in response to specific events, such as a data update in the primary source.

• Mechanism: The application issues a command (e.g., cache.delete(key)) when source data changes.
• Pros: Guarantees fresh data is fetched immediately after an update, enabling strong consistency.
• Cons: Requires a publish-subscribe mechanism or hooks into the data layer to know when to invalidate.
• Use Case: An agent caching user profile data; the cache is invalidated the moment the user updates their profile.

This strategy ties cache validity to a version identifier (e.g., an ETag, hash, or timestamp). The cache key includes the version. When the source data changes, its version changes, making all old keys obsolete.

• How it works: The agent stores the result of fetch_data(user_id, data_version). A new version generates a new cache key, causing a miss for old keys.
• Pros: Granular and deterministic. Naturally handles distributed caches.
• Cons: Can lead to cache pollution if versions change frequently without cleanup.
• Use Case: Caching the results of database queries where the schema or underlying data version is known.

These are cache consistency patterns that inherently manage invalidation during write operations.

• Write-Through: Data is written synchronously to both the cache and the primary store. The cache always has the fresh value, but writes are slower.
• Write-Behind (Write-Back): Data is written to the cache immediately and asynchronously queued for the primary store. This offers write performance but risks data loss if the cache fails before the flush.
• Invalidation Role: They ensure the cache is updated on writes, preventing stale reads for updated data. They are often combined with TTL for other cleanup.

The cache subscribes to a stream of change events (e.g., from a database CDC log or a message bus) and invalidates relevant entries as events are published.

• Architecture: Requires an event producer (primary database), a message broker (Kafka, Redis Pub/Sub), and cache consumers.
• Pros: Enables very low latency between source update and cache invalidation, supporting near-real-time consistency.
• Cons: Complex infrastructure; requires careful mapping of events to cache keys.
• Use Case: High-frequency trading agents where cached market data must be invalidated within milliseconds of a new trade.

This strategy introduces randomness to prevent cache stampedes. Instead of all items expiring at exactly the same time, their expiration is jittered.

• Mechanism: A base TTL is augmented with a random, small offset (e.g., TTL ± random(10%)).
• Pros: Smoothes out load on the primary data source, preventing thundering herds after mass expiration.
• Cons: Some items may be invalidated slightly earlier than necessary.
• Related Pattern: Often used with stale-while-revalidate, where stale data can be served while a background refresh occurs.

Time-To-Live (TTL) is a proactive cache invalidation policy that defines the maximum duration a cached item is considered valid before it is automatically expired and must be refreshed. It is the most common method for ensuring data freshness without explicit invalidation signals.

• Mechanism: Each cache entry is stamped with an expiration timestamp. The cache middleware checks this timestamp on read; if expired, the data is treated as a cache miss.
• Use Case: Ideal for data with predictable update cycles (e.g., hourly weather reports, daily currency rates).
• Trade-off: Simplicity vs. potential staleness. A short TLL ensures freshness but increases load on the primary source; a long TLL improves hit rates but risks serving outdated data.

Cache eviction is the process of removing items from a cache to free up space for new entries, governed by policies like LRU or LFU. While invalidation marks data as stale due to content changes, eviction removes data due to capacity constraints.

• Primary Driver: Cache capacity limits, not data freshness.
• Common Policies:
  • Least Recently Used (LRU): Evicts the item not accessed for the longest time.
  • Least Frequently Used (LFU): Evicts the item with the fewest accesses over a period.
  • First-In-First-Out (FIFO): Evicts the oldest item, regardless of use.
• Interaction with Invalidation: An evicted item is simply removed; if it was still valid, a subsequent request will result in a cache miss and recomputation.

Cache consistency refers to the property that ensures the data stored in a cache accurately reflects the data in the primary source. Invalidation is the primary mechanism to enforce consistency.

• Strong Consistency: Guarantees that a read returns the most recent write. Requires synchronous invalidation or write-through caching, which can impact performance.
• Eventual Consistency: A model where updates are propagated asynchronously. Caches may serve stale data for a short window until an invalidation message is processed. This is common in distributed agent systems.
• Challenge in Multi-Agent Systems: Maintaining consistency across caches in different agent sessions or nodes requires a distributed invalidation protocol (e.g., publish-subscribe for invalidation messages).

A write-through cache is a caching pattern where data is written synchronously to both the cache and the primary data source. This pattern simplifies invalidation because the cache is updated simultaneously with the source of truth.

• Invalidation Implication: Since the cache is updated on write, subsequent reads are guaranteed to be fresh (strong consistency). There is no separate invalidation step for the written data.
• Performance Trade-off: Write latency is higher because it must complete two writes. This is acceptable for systems where read performance is critical and write volume is moderate.
• Contrast with Write-Behind: Write-behind caches write to the source asynchronously, creating a window where the cache holds the only updated copy, complicating invalidation for other readers.

A cache stampede (or thundering herd) is a performance degradation scenario where the simultaneous expiration (invalidation) of many cache items causes a sudden surge of requests to hit the primary data source, potentially overwhelming it.

• Cause: Poorly coordinated TTL expirations or a broadcast invalidation event affecting many popular items.
• Mitigation Strategies:
  • Staggered Expiration: Add jitter (random variation) to TTL values to spread out recomputation.
  • Locking/Mutex: Have the first request after expiration compute the new value and block other requests until it's done, allowing them to then read from the refreshed cache.
  • Background Refresh: Use patterns like stale-while-revalidate to serve stale data while one request updates the cache in the background.

A semantic cache stores the results of previous LLM inferences or computations based on the meaning or intent of a query, rather than an exact string match. Invalidation for semantic caches is more complex than for key-value caches.

• Cache Key Challenge: The key is a vector embedding of the query's semantics. Similar queries (not identical) can result in a cache hit.
• Invalidation Challenge: Determining when a cached semantic result is stale requires understanding if the underlying knowledge or context has changed in a way that affects the meaning of potential future queries.
• Approach: Often relies on metadata tagging of cached items with data source versions or knowledge cut-off dates. Invalidation can be triggered when source data is updated, invalidating all semantic entries derived from that source.