Cache Hierarchy (L1/L2/L3)

Acts as a secondary buffer between the fast L1 and the larger L3 or main memory.

• Speed: Slower than L1, with latencies in the 10-20 cycle range.
• Size: Larger than L1, typically 256 KB to 1 MB per core.
• Structure: Often unified (holding both instructions and data).
• Scope: May be private per core or shared between a small cluster of cores (e.g., a pair). It services requests that miss in the L1 cache.

The largest and slowest cache level within the CPU package, serving as a shared pool for all cores.

• Speed: Higher latency, often 30-50 cycles.
• Size: Much larger, ranging from 8 MB to over 100 MB on modern server CPUs.
• Structure: Always unified and shared across all CPU cores.
• Purpose: Reduces traffic to the main memory (RAM) by intercepting requests that miss in the L2 caches. It is crucial for core-to-core data sharing and multi-threaded performance.

A critical architectural choice defining the relationship between cache levels.

• Inclusive Cache: Data present in a higher-level cache (e.g., L1) is guaranteed to also be present in all lower-level caches (e.g., L2, L3). This simplifies cache coherence but reduces effective capacity.
• Exclusive Cache: Data is guaranteed to reside in only one level of the hierarchy at a time. This maximizes total effective cache capacity but complicates coherence protocols.
• Most Common: Modern Intel CPUs typically use inclusive L3 caches, while AMD Zen architectures have used exclusive L3 designs.

A hardware mechanism that maintains a single, consistent view of memory across all cores' private caches.

• Problem: When Core A modifies data in its private L1 cache, Core B's cached copy becomes stale.
• Solution: Protocols like MESI (Modified, Exclusive, Shared, Invalid) use states and messages over the CPU interconnect to track line ownership and invalidate stale copies.
• Importance: This protocol is essential for correct execution in multi-core systems and is managed transparently by the hardware, with the shared L3 cache often acting as a coherence hub.

An automated storage management technique that dynamically moves data between different classes of memory or storage media based on access patterns and predefined policies. It operationalizes the memory hierarchy.

• Hot vs. Cold Data: Frequently accessed ('hot') data is kept in faster, more expensive tiers (e.g., RAM, NVMe). Infrequently accessed ('cold') data is relegated to slower, cheaper tiers (e.g., HDD, object storage).
• Policy-Driven: Movement can be based on recency, frequency, or predictive prefetching.
• Application: Critical for cost-effective management of large-scale vector databases and knowledge graphs, ensuring low-latency access to active working sets.

A computational principle stating that memory accesses are not random but tend to cluster, which is exploited by cache hierarchies for performance gains. There are two key types:

• Temporal Locality: Recently accessed data is likely to be accessed again soon. Caches exploit this by retaining recently used items.
• Spatial Locality: Data located near recently accessed data is likely to be accessed soon. Caches exploit this by fetching data in blocks (cache lines).
• Agentic Analogy: An agent working on a multi-step task will repeatedly reference the same set of documents (temporal locality) and related concepts within those documents (spatial locality), making caching highly effective.

A performance optimization technique where a memory system predicts future data needs and loads that data into a cache before it is explicitly requested by the processor or agent.

• Pattern-Based: Uses algorithms to detect sequential or strided access patterns in memory addresses or data streams.
• Agentic Application: An advanced agent might prefetch likely-needed knowledge from a long-term store based on the initial steps of a plan, similar to a CPU prefetching the next cache line.
• Risk/Reward: Correct prefetching hides memory latency; incorrect prefetching wastes bandwidth and cache space with useless data.

A memory design for multiprocessor systems where the memory access time depends on the memory location relative to the processor. This creates a hierarchy of memory 'distance' within a single machine.

• Local vs. Remote Memory: A CPU can access its directly attached RAM ('local' memory) faster than RAM attached to another CPU ('remote' memory) across an interconnect.
• Performance Implication: Software (OS, agents) must be 'NUMA-aware' to schedule threads and allocate memory optimally, minimizing costly remote accesses. Poor management can negate the benefits of multiple cores.
• Large-System Relevance: Critical for performance in high-core-count servers running multi-agent systems or large language model inference.