Memory Hierarchy

This principle states that programs tend to access data and instructions in predictable, clustered patterns. Spatial locality refers to the tendency to access data items whose addresses are near recently referenced items. Temporal locality refers to the tendency to access the same data items repeatedly over a short time period. Memory hierarchies exploit this by storing recently and nearby used data in faster cache levels.

• Example: A loop iterating over an array exhibits strong spatial locality. A frequently called function variable exhibits strong temporal locality.

This is the fundamental constraint driving hierarchical design. No single memory technology can simultaneously optimize for all three properties.

• Speed (Latency/Bandwidth): Faster access (e.g., CPU registers, SRAM cache) requires more expensive, power-hungry circuits.
• Size (Capacity): Larger storage (e.g., DRAM, SSDs) is cheaper per bit but introduces higher latency.
• Cost per Bit: The hierarchy balances performance needs with economic feasibility, placing small amounts of fast memory close to the processor and larger, slower storage further away. In agentic systems, this manifests as a trade-off between a fast working memory buffer (small, expensive context) and a large vector memory store (slower, cheaper retrieval).

These are critical consistency properties in multi-level cache hierarchies.

• Inclusion Policy: Dictates the relationship between contents of different levels. A common policy is inclusive caching, where all data in a smaller, faster cache (L1) is also present in a larger, slower cache (L2). This simplifies coherence but may waste space.
• Coherence Protocol: Ensures all copies of a data block (e.g., in multiple CPU caches) are consistent. Protocols like MESI (Modified, Exclusive, Shared, Invalid) manage state transitions to guarantee that a read operation returns the most recently written value, even in multi-core systems. For agentic systems, similar principles apply to maintaining consistency between a short-term memory cache and a persistent memory layer.

These are performance optimization techniques that leverage the principle of locality.

• Block Transfer (Cache Line): Data is moved between hierarchy levels in fixed-size blocks (e.g., 64-byte cache lines), not as individual bytes. This amortizes the high latency of accessing slower memory by fetching spatially local data that is likely to be used soon.
• Prefetching: The memory system proactively loads data into a faster level before it is explicitly requested by the processor or agent. Predictors analyze access patterns (sequential, strided) to hide memory latency. In AI contexts, this is analogous to an agent pre-loading relevant context from a knowledge graph memory into its working buffer based on the task flow.

This is a core operating system mechanism that abstracts physical memory resources, providing each process with a uniform, isolated address space. It relies heavily on the memory hierarchy (RAM as physical memory, disk as swap space).

• Paging: Memory is divided into fixed-size pages. A page table, managed by the Memory Management Unit (MMU), maps virtual pages to physical frames in RAM.
• Translation Lookaside Buffer (TLB): A special cache for page table entries to accelerate address translation.
• Swapping: Idle pages can be written to disk (swap file) to free RAM, extending the effective memory hierarchy. This concept is mirrored in agent systems where less frequently accessed memories are tiered to slower, cheaper storage.

A memory architecture for multiprocessor systems where access time to shared memory depends on the memory location relative to the requesting processor.

• Local Memory: Memory physically attached to a processor's socket is faster to access.
• Remote Memory: Memory attached to another socket has higher latency due to interconnect traversal.
• Implication: Software must be aware of data placement (memory locality) to avoid performance penalties. This principle scales to distributed multi-agent systems, where accessing memory for multi-agent systems in a remote node is slower than accessing local agent state.

A dynamic storage management technique that automatically moves data between different classes of memory or storage media based on access patterns and policies. Hot data (frequently accessed) resides in fast, expensive media like DRAM or NVMe. Cold data (infrequently accessed) is moved to slower, cheaper media like SATA SSDs or HDDs. This optimizes cost-performance trade-offs in large-scale systems, similar to how an agent might prioritize recent experiences in working memory while archiving older ones.

A short-term, high-speed memory component in an agentic system that temporarily holds and manipulates information relevant to the current task. It is the cognitive equivalent of a CPU's registers and L1 cache.

• Function: Maintains the immediate context for reasoning, planning, and tool execution.
• Characteristics: Limited capacity, volatile, and extremely fast access.
• Example: In a coding agent, the working buffer holds the current file being edited, the specific function being written, and the immediate user instructions.

A persistent, high-capacity memory component designed for the durable storage of knowledge, experiences, and skills over extended timeframes. It is analogous to a computer's main memory (RAM) and disk storage.

• Function: Archives learned facts, user preferences, historical interactions, and procedural knowledge.
• Characteristics: High capacity, persistent, and slower to access than working memory.
• Implementation: Often built using vector databases for semantic search or knowledge graphs for structured reasoning.

A storage system that represents information as high-dimensional vectors (embeddings) to enable efficient similarity-based search and retrieval. It is a core technology for implementing semantic long-term memory in AI agents.

• Mechanism: Text, images, or other data are converted into dense vector embeddings via a model. Similar vectors represent semantically similar concepts.
• Retrieval: Given a query embedding, the store performs a nearest-neighbor search to find the most relevant stored memories.
• Use Case: Allows an agent to recall past conversations or documents related to the current topic, even without exact keyword matches.