Short-Term Memory Cache

Caches are constrained by memory size, necessitating intelligent eviction policies to manage what stays in the fast-access layer. Common policies include:

• LRU (Least Recently Used): Discards the oldest accessed items.
• TTL (Time-To-Live): Automatically expires data after a fixed duration.
• LFU (Least Frequently Used): Evicts items with the lowest access count. This limited capacity forces a hierarchical memory design, where only the "hottest" data resides in the cache, with less-frequently accessed information relegated to slower, larger stores.

It acts as a context window manager for agents powered by Large Language Models (LLMs). Since LLMs have fixed context windows, the cache holds the immediate operational context—such as recent tool outputs, user queries, and system prompts—that must be included in the next inference call. This prevents the need to repeatedly retrieve the same data from slower sources, a process critical for managing Context Window limits efficiently.

A major engineering challenge is cache invalidation—knowing when cached data is stale and must be refreshed or purged. In agentic systems, this occurs when:

• Underlying knowledge sources are updated (e.g., a database record changes).
• The agent's own reasoning invalidates a prior assumption.
• A multi-agent system signals a state change. Poor invalidation leads to coherence problems, where the agent acts on outdated information, breaking the consistency of its interaction with the world.

A short-term cache is not a standalone system; it is the fastest tier in a Memory Hierarchy. Its effectiveness depends on seamless integration with other layers:

• Read-Through/Write-Through Patterns: Automatically loading from or writing to a Persistent Memory Layer.
• Warm-Up Sequences: Pre-loading anticipated data at agent startup.
• Fallback to Slower Stores: On a cache miss, querying a Vector Memory Store or Knowledge Graph Memory. This layered approach, analogous to CPU Cache Hierarchy (L1/L2/L3), balances the need for speed with the need for comprehensive knowledge access.

The principle that memory accesses tend to cluster, which is exploited by caching to improve performance. Two key types:

• Temporal Locality: Recently accessed data is likely to be accessed again soon.
• Spatial Locality: Data located near recently accessed data is likely to be accessed next. An agent's short-term memory cache is designed explicitly to exploit temporal locality, holding recent interactions or tool outputs for immediate reuse, drastically reducing latency in loops or repetitive queries.

A short-term, high-speed memory component in an agentic system that temporarily holds and manipulates information relevant to the current task or cognitive operation. It is the active "scratchpad" for reasoning. While a short-term memory cache is optimized for fast retrieval of recent outputs, a working memory buffer is often more tightly integrated with the agent's planning and execution loops, holding the immediate state of a decomposed task.

A specialized, high-speed cache that stores recent translations of virtual memory addresses to physical memory addresses. It is a critical hardware component for virtual memory systems. Every memory access requires an address translation; the TLB acts as a short-term cache for these mappings, avoiding slower page table walks. This is a canonical example of a cache designed to accelerate a repetitive, latency-sensitive operation—directly parallel to caching recent agent interactions.

The algorithm that determines which data is removed from a cache when it becomes full. Common policies include:

• Least Recently Used (LRU): Evicts the data unused for the longest time.
• First-In, First-Out (FIFO): Evicts the oldest data.
• Random Replacement: Evicts a random entry. The choice of eviction policy (e.g., LRU for an agent's conversation history) fundamentally determines the hit rate and performance characteristics of any short-term memory cache.