Memory Access Pattern

Spatial locality is the principle that if a particular memory location is accessed, nearby memory locations are also likely to be accessed in the near future. This is fundamental to cache line design and prefetching effectiveness.

• Cache Line Utilization: Efficient patterns access all data within a fetched cache line before it is evicted.
• Strided Access: A common pattern where data is accessed at fixed intervals (stride). A stride of 1 exhibits perfect spatial locality, while large, non-unit strides waste bandwidth.
• Example: Processing a multi-dimensional tensor in row-major order versus column-major order results in vastly different spatial locality and performance on NPUs.

Temporal locality is the principle that if a memory location is accessed, it is likely to be accessed again soon. This drives the effectiveness of caching hierarchies by keeping frequently used data in fast, on-chip memory.

• Reuse Distance: The number of distinct data elements accessed between consecutive uses of the same element. Short reuse distances enable high cache hit rates.
• Working Set: The set of data actively used within a time window must fit within the available cache capacity to avoid thrashing.
• NPU Impact: Kernel fusion and tiling are compiler optimizations explicitly designed to increase temporal locality for intermediate tensors in neural networks.

Access granularity refers to the size of each memory transaction, while alignment requires data addresses to be multiples of this granularity for optimal performance.

• Coalesced Access: On NPUs and GPUs, concurrent threads should access contiguous, aligned memory blocks to merge requests into a single, wide transaction. Uncoalesced access serializes transactions, crippling bandwidth.
• Hardware Mandate: Most high-bandwidth memory (HBM) interfaces and on-chip interconnects require naturally aligned accesses (e.g., 32-byte, 128-byte) for peak throughput.
• Performance Penalty: Misaligned or sub-granular accesses trigger read-modify-write cycles or partial transactions, significantly increasing effective latency.

Modern memory systems are interleaved across multiple physical banks to increase aggregate bandwidth. Access patterns must be structured to avoid conflicts.

• Bank Conflict: Occurs when multiple concurrent requests target the same memory bank, causing serialization. Requests to different banks can proceed in parallel.
• Address Mapping: The mapping of linear addresses to banks and rows is hardware-defined. Patterns with a stride equal to a power-of-two often cause severe bank conflicts.
• NPU/GPU Focus: Achieving high occupancy requires scheduling thousands of threads whose collective access pattern avoids bank conflicts in shared memory or scratchpad SRAM.

Prefetching relies on predictable access patterns to hide memory latency by moving data into cache before it is demanded. Unpredictable patterns defeat prefetchers.

• Regular Strides: Simple linear or constant-stride patterns are easily predicted by hardware stream prefetchers.
• Indirect Accesses: Patterns involving pointer chasing (e.g., graph traversals, sparse matrices) are inherently unpredictable, often requiring software-managed prefetch hints or different algorithms.
• Compiler Role: Advanced NPU compilers analyze loop nests to insert software prefetch instructions for complex but analyzable patterns.

The balance between read and write operations, and the need for coherence, defines the required memory subsystem capabilities and protocol overhead.

• Read-Dominant: Most ML inference workloads are heavily read-dominant (weights, activations), optimizing for high read bandwidth.
• Write-Back vs. Write-Through: Write-back policies (deferring writes) benefit write-heavy operations but add coherence complexity. NPU scratchpads often use explicit software management to avoid hardware coherence overhead.
• Non-Temporal Writes: For data written once and not re-read soon (e.g., final output), streaming or non-temporal store instructions bypass caches to prevent pollution and save bandwidth.

Coalesced access is a fundamental optimization where consecutive threads in a warp or wavefront access consecutive memory locations. This allows the memory controller to combine multiple requests into a single, wider transaction.

• Mechanism: On an NPU, a 128-byte cache line request can satisfy 32 threads each reading a 4-byte float, if their accesses are contiguous.
• Impact: Dramatically reduces the number of memory transactions, maximizing effective bandwidth.
• Anti-Pattern: Strided or random access prevents coalescing, serializing requests and crippling throughput.

Tiling (or blocking) is a data layout transformation that decomposes a large dataset into smaller sub-blocks (tiles) that fit entirely within a fast, on-chip memory like an NPU's scratchpad memory or L1 cache.

• Process: A computation on a large matrix is broken into operations on smaller sub-matrices. Each tile is loaded once, used for multiple computations, and then written back.
• Benefit: Exploits temporal locality by reusing data within the tile, and spatial locality via contiguous access within the block.
• Use Case: Essential for optimizing Convolutional Neural Network (CNN) and General Matrix Multiply (GEMM) kernels on NPUs.

Software prefetching involves explicitly inserting instructions to move data from slower memory (e.g., DRAM) into a faster cache level before it is needed by the compute units, thereby hiding memory latency.

• Control: Unlike hardware prefetching, which reacts to patterns, software prefetch gives the programmer direct control over data movement timing.
• NPU Application: Used in NPU kernels to pipeline computation and data transfer. While one tile is being processed, the next tile is prefetched into the scratchpad.
• Challenge: Requires accurate prediction of future accesses and careful scheduling to avoid polluting the cache with unused data.

This technique involves restructuring how data is arranged in memory to match the NPU's optimal access pattern, often converting Array-of-Structures (AoS) to Structure-of-Arrays (SoA).

• AoS Problem: struct {float x, y, z;} points[N]; If a kernel only needs the x field, accesses are strided by 12 bytes, preventing coalescing.
• SoA Solution: struct {float x[N], y[N], z[N];} All x values are contiguous, enabling perfectly coalesced reads.
• NPU Consideration: NPU compilers often perform these transformations automatically, but manual control can be critical for irregular data structures or custom operations.

Shared memory or scratchpad memory on NPUs is typically partitioned into independent banks that can be accessed simultaneously. A bank conflict occurs when two or more threads in the same instruction issue attempt to access different addresses within the same memory bank, forcing serialized access.

• Cause: If shared memory has 32 banks (address interleaved), threads with a stride that is a multiple of 32 will collide.
• Optimization: Pad array dimensions or carefully choose access strides to ensure threads map to unique banks.
• Performance Impact: Severe bank conflicts can reduce shared memory bandwidth to a single access per cycle, nullifying its performance benefit.

Vectorized memory operations use wide Single Instruction, Multiple Data (SIMD) or vector instructions to load or store multiple data elements with a single instruction. This increases instruction-level parallelism and reduces instruction fetch overhead.

• Hardware Support: Modern NPUs have wide vector units (e.g., 128-bit, 256-bit) supporting load/store instructions for multiple contiguous elements.
• Compiler Role: The compiler attempts to auto-vectorize loops, but programmers can use intrinsics (e.g., vld4q_f32 for ARM NEON) for explicit control.
• Prerequisite: Requires memory alignment to the vector width (e.g., 16-byte alignment for 128-bit loads) for maximum performance and correctness on many architectures.

A principle of program behavior where accessing a memory location makes it likely that nearby locations will be accessed soon. This is the foundation for efficient cache line utilization and hardware prefetching. For example, iterating through a contiguous array exhibits high spatial locality, as each access is to an adjacent address. NPU compilers optimize for this by structuring data layouts and access patterns to maximize sequential, strided accesses that hardware prefetchers can predict.

A principle of program behavior where a recently accessed memory location is likely to be accessed again in the near future. This is the core rationale for caching. In NPU kernels, reusing a weight tensor or an activation value across multiple operations exhibits temporal locality. Performance optimization involves identifying these hot data items and explicitly managing their placement in fast, on-chip memory (like scratchpad SRAM) to minimize repeated, costly fetches from global DRAM.

The maximum rate at which data can be transferred between a processor (like an NPU) and its memory subsystem, measured in bytes per second (e.g., GB/s). It is a key throughput metric. An inefficient memory access pattern that causes scattered, non-coalesced reads can saturate available bandwidth long before the compute units are fully utilized, creating a bandwidth-bound scenario. High-performance kernels are designed to structure accesses to achieve near-peak theoretical bandwidth.

The time delay between issuing a memory request and receiving the data, typically measured in clock cycles or nanoseconds. Latency is distinct from bandwidth; latency is about the time to first byte, while bandwidth is about bytes per second. Irregular access patterns with poor locality cause frequent cache misses, exposing the full, high latency of DRAM (often hundreds of cycles). Techniques like prefetching and software pipelining are used to hide this latency.

A critical optimization for parallel architectures (GPUs, NPUs) where concurrent threads (or processing elements) access a contiguous block of memory simultaneously. This allows the memory controller to combine multiple requests into a single, wider transaction, maximizing effective bandwidth. The opposite—non-coalesced or scattered access—results in many small, inefficient transactions. Writing kernels with stride-1 access patterns across parallel threads is a fundamental performance requirement.

A hardware or software technique that predicts future memory accesses and proactively loads data into a cache or buffer before it is explicitly requested by the program, aiming to hide memory latency. Hardware prefetchers detect simple patterns like sequential or constant-stride accesses. For complex, data-dependent patterns, software prefetching using explicit compiler intrinsics can be inserted to bring data into cache well ahead of its use, ensuring compute units are not stalled.