Memory Locality

A specific, strong form of spatial locality where data elements are accessed in a predictable, linear order (e.g., array traversal). This allows for highly accurate prefetching.

• Mechanism: Hardware prefetchers detect strided access patterns and automatically fetch the next anticipated cache lines before the CPU explicitly requests them.
• Example in AI: Streaming through a training dataset batch, performing a forward pass through layers in order, or iterating over tokens in a text sequence all demonstrate sequential locality. This predictability is key for optimizing data pipeline throughput.

A pattern where memory accesses occur at fixed, regular intervals (strides). This is common in matrix and tensor operations.

• Mechanism: Advanced hardware prefetchers can detect constant-stride patterns (e.g., accessing every 4th element) and prefetch accordingly. Poor handling of strided access can lead to cache thrashing.
• Example in AI: Accessing a column of a row-major matrix involves a large stride equal to the row length. Operations like transposition or certain convolutional layers exhibit strided patterns. Libraries like cuBLAS and TensorFlow/XLA optimize kernel execution to minimize the negative performance impact of strided accesses.

The predictability in the flow of execution (instruction access), rather than data access. It refers to the tendency of program execution to cluster in specific regions of the instruction stream.

• Mechanism: Branch predictors in CPUs use historical patterns to guess the outcome of conditional jumps (if/else, loops), enabling speculative execution and keeping the instruction pipeline full.
• Example in AI: The control flow within a neural network's inference graph or the decision logic in an agentic reasoning loop (e.g., if condition X is met, then call tool Y) exhibits branch locality. Predictable branches lead to higher Instructions Per Cycle (IPC).

The set of memory addresses a process references during a specific time interval (its working set) tends to be smaller than its total addressable memory. Performance is good when the working set fits in fast cache.

• Mechanism: If the working set exceeds the cache capacity, capacity misses occur, forcing frequent evictions and reloads from main memory, drastically slowing performance.
• Example in AI: The context window of a Large Language Model defines a working set of tokens. If an agentic workflow requires reasoning over a context larger than the model's window, performance degrades. Techniques like hierarchical memory or vector database retrieval are used to manage the effective working set presented to the core model.

A hardware component that manages memory access, performing critical tasks that rely on and affect locality. Its primary functions include:

• Virtual-to-Physical Address Translation: Maps process virtual addresses to physical RAM. This enables features like paging, which can disrupt locality if not managed well.
• Translation Lookaside Buffer (TLB) Management: The TLB is a cache for page table entries, exploiting temporal locality in address translations. A TLB miss incurs a significant penalty.
• Memory Protection: Enforces access permissions, ensuring processes cannot corrupt each other's memory spaces. The MMU's efficiency is heavily dependent on the locality of memory references within a process.

A multiprocessor architecture where memory access time depends on the memory location's proximity to the requesting processor core. This creates a locality of access at the system level.

• Local Memory: Memory attached directly to a processor's socket. Access is fast (low latency).
• Remote Memory: Memory attached to another processor's socket. Access is slower due to inter-socket communication.
• NUMA Awareness: Operating systems and applications optimized for NUMA will try to allocate memory and schedule threads to maximize local accesses, directly applying the principle of memory locality to system design. Poor NUMA management leads to severe performance degradation.

A storage management technique that automatically moves data between different classes of memory or storage media based on access patterns, effectively implementing locality-aware data placement.

• Hot Data: Frequently accessed data is kept in fast, expensive media (e.g., DRAM, NVMe).
• Cold Data: Infrequently accessed data is relegated to slower, cheaper media (e.g., SATA SSDs, HDDs).
• Policy Engine: Uses metrics like access frequency and recency to promote/demote data between tiers. This is a direct application of temporal locality at the storage system level, optimizing cost-performance trade-offs.

In agentic and cognitive architectures, this is the analog to a CPU cache—a short-term, high-speed memory that holds information relevant to the current task. Its design and effectiveness are governed by locality principles.

• Temporal Locality: Recently generated thoughts, tool outputs, or user messages are kept immediately accessible for the next step in a reasoning loop.
• Spatial/Contextual Locality: Related pieces of information (e.g., parameters for a function call, supporting context for a query) are co-located within the buffer.
• Eviction Policies: When full, the buffer uses policies (e.g., LRU - Least Recently Used) that assume temporal locality will hold, evicting the least recently used information first.