Memory Locality

The principle that data accessed recently is likely to be accessed again in the near future. This is the foundational concept behind caching.

• Mechanism: When a memory address is read or written, its value is stored in a fast cache. Subsequent accesses to the same address can be served from the cache, avoiding slower main memory access.
• Example in AI: The weights of a frequently used layer in a neural network exhibit high temporal locality during inference. The KV (Key-Value) cache in transformer-based LLMs exploits this to store previously computed attention keys and values, avoiding redundant computation for tokens seen earlier in the sequence.

The principle that if a particular memory location is accessed, nearby memory locations are likely to be accessed soon. This enables efficient prefetching and cache line design.

• Mechanism: Memory systems fetch data in blocks (cache lines), not individual bytes. Accessing address X causes the entire block containing X to be loaded into the cache.
• Example in AI: Processing a dense vector or a contiguous block of a weight matrix exhibits perfect spatial locality. When an embedding vector is fetched for a similarity search in a vector database, the entire vector (a contiguous block of floats) is loaded at once, making the subsequent comparisons extremely fast.

A specific, strong form of spatial locality where data elements are accessed in a predictable, linear order (e.g., array traversal). This allows for highly accurate prefetching.

• Mechanism: Hardware prefetchers detect strided access patterns and automatically fetch the next anticipated cache lines before the CPU explicitly requests them.
• Example in AI: Streaming through a training dataset batch, performing a forward pass through layers in order, or iterating over tokens in a text sequence all demonstrate sequential locality. This predictability is key for optimizing data pipeline throughput.

A pattern where memory accesses occur at fixed, regular intervals (strides). This is common in matrix and tensor operations.

• Mechanism: Advanced hardware prefetchers can detect constant-stride patterns (e.g., accessing every 4th element) and prefetch accordingly. Poor handling of strided access can lead to cache thrashing.
• Example in AI: Accessing a column of a row-major matrix involves a large stride equal to the row length. Operations like transposition or certain convolutional layers exhibit strided patterns. Libraries like cuBLAS and TensorFlow/XLA optimize kernel execution to minimize the negative performance impact of strided accesses.

The predictability in the flow of execution (instruction access), rather than data access. It refers to the tendency of program execution to cluster in specific regions of the instruction stream.

• Mechanism: Branch predictors in CPUs use historical patterns to guess the outcome of conditional jumps (if/else, loops), enabling speculative execution and keeping the instruction pipeline full.
• Example in AI: The control flow within a neural network's inference graph or the decision logic in an agentic reasoning loop (e.g., if condition X is met, then call tool Y) exhibits branch locality. Predictable branches lead to higher Instructions Per Cycle (IPC).

The set of memory addresses a process references during a specific time interval (its working set) tends to be smaller than its total addressable memory. Performance is good when the working set fits in fast cache.

• Mechanism: If the working set exceeds the cache capacity, capacity misses occur, forcing frequent evictions and reloads from main memory, drastically slowing performance.
• Example in AI: The context window of a Large Language Model defines a working set of tokens. If an agentic workflow requires reasoning over a context larger than the model's window, performance degrades. Techniques like hierarchical memory or vector database retrieval are used to manage the effective working set presented to the core model.

A multi-level structure of small, fast memory caches integrated into a CPU to exploit memory locality. Data is staged through levels (L1, L2, L3) based on recency and frequency of use.

• L1 Cache: Fastest, smallest, private per core. Holds instructions and data with the highest temporal locality.
• L2 Cache: Larger, slower, often shared between cores. Catches accesses that miss L1.
• L3 Cache: Largest, slowest, shared across all cores. Serves as a final buffer before accessing main RAM. This hierarchy directly implements the principle of locality to minimize average memory access latency.

A hardware or software optimization that predicts future memory addresses and loads data into cache before the processor explicitly requests it. It is a proactive technique to hide memory latency by exploiting predictable access patterns.

• Spatial Prefetching: Loads adjacent memory blocks (following spatial locality), common for array traversals.
• Stream Prefetching: Detects sequential access patterns and fetches ahead in the stream.
• Software Prefetching: Uses special CPU instructions (e.g., prefetch) inserted by the compiler or programmer to hint at future accesses. Effective prefetching relies on accurate prediction of locality patterns.

A hardware component that manages memory access, performing critical tasks that rely on and affect locality. Its primary functions include:

• Virtual-to-Physical Address Translation: Maps process virtual addresses to physical RAM. This enables features like paging, which can disrupt locality if not managed well.
• Translation Lookaside Buffer (TLB) Management: The TLB is a cache for page table entries, exploiting temporal locality in address translations. A TLB miss incurs a significant penalty.
• Memory Protection: Enforces access permissions, ensuring processes cannot corrupt each other's memory spaces. The MMU's efficiency is heavily dependent on the locality of memory references within a process.

A multiprocessor architecture where memory access time depends on the memory location's proximity to the requesting processor core. This creates a locality of access at the system level.

• Local Memory: Memory attached directly to a processor's socket. Access is fast (low latency).
• Remote Memory: Memory attached to another processor's socket. Access is slower due to inter-socket communication.
• NUMA Awareness: Operating systems and applications optimized for NUMA will try to allocate memory and schedule threads to maximize local accesses, directly applying the principle of memory locality to system design. Poor NUMA management leads to severe performance degradation.

A storage management technique that automatically moves data between different classes of memory or storage media based on access patterns, effectively implementing locality-aware data placement.

• Hot Data: Frequently accessed data is kept in fast, expensive media (e.g., DRAM, NVMe).
• Cold Data: Infrequently accessed data is relegated to slower, cheaper media (e.g., SATA SSDs, HDDs).
• Policy Engine: Uses metrics like access frequency and recency to promote/demote data between tiers. This is a direct application of temporal locality at the storage system level, optimizing cost-performance trade-offs.

In agentic and cognitive architectures, this is the analog to a CPU cache—a short-term, high-speed memory that holds information relevant to the current task. Its design and effectiveness are governed by locality principles.

• Temporal Locality: Recently generated thoughts, tool outputs, or user messages are kept immediately accessible for the next step in a reasoning loop.
• Spatial/Contextual Locality: Related pieces of information (e.g., parameters for a function call, supporting context for a query) are co-located within the buffer.
• Eviction Policies: When full, the buffer uses policies (e.g., LRU - Least Recently Used) that assume temporal locality will hold, evicting the least recently used information first.