Memory Latency

Access time, often specified as Column Address Strobe (CAS) latency (tCAS), is the core delay between a memory controller issuing a read command with a column address and the first bit of data appearing at the output pins of the DRAM chip. This is the fundamental latency of the memory cell array itself.

• Mechanism: After a row is activated (tRAS), the controller sends a column address. The tCAS delay encompasses the time to decode the address, select the correct column, and sense the data from the bitlines.
• Typical Values: For DDR4 memory, tCAS can range from 14 to 22 clock cycles. For High Bandwidth Memory (HBM) used with NPUs, latencies are lower but follow the same principle.
• Impact: This is the irreducible minimum latency for a random access once a row is open. Reducing tCAS directly improves random access performance.

DRAM is organized in banks, rows, and columns. Accessing data requires managing the state of these rows, adding significant overhead to latency.

• Row Activation (tRAS): The delay to open (activate) a row in a bank, copying its contents into the row buffer. This is necessary before any column in that row can be read.
• Precharge (tRP): The delay to close an open row and prepare the bank for a new activation. A bank must be precharged before a different row can be activated.
• The Penalty: A row miss (accessing a closed row) incurs tRAS + tCAS. A row conflict (accessing a different row in an open bank) incurs tRP + tRAS + tCAS, which can be over 100ns. Optimizing access patterns for spatial locality to stay within an open row is critical for NPU kernels.

The wires connecting the memory controller to the DRAM modules (the command/address bus) are a shared resource. Switching the direction of traffic or changing command types introduces small but non-zero delays.

• Bus Turnaround Time: The idle cycles required when switching the bus from driving read data to driving a new command, or vice-versa, to avoid electrical contention.
• Write-to-Read Delay (tWTR): The minimum delay required between issuing a write command and a subsequent read command to the same DRAM rank. This allows write data to be fully captured internally before the bus is reconfigured for a read.
• Read-to-Write Delay (tRTW): The analogous delay when switching from read to write commands.
• System Impact: These protocol-level timings constrain command scheduling efficiency. An optimal memory controller reorders requests to minimize these turnaround penalties, which is vital for maintaining high bandwidth and low effective latency in NPU workloads.

Within a modern System-on-Chip (SoC) or NPU, the physical distance between a compute core and the memory controller introduces propagation delay. The on-chip network that routes the request and response adds further latency.

• Hop Latency: The time for a packet (memory request) to traverse a router or switch within the Network-on-Chip (NoC). Each hop adds cycles of delay.
• Contention Delay: When multiple cores (e.g., NPU tensor cores) issue requests simultaneously, they may contend for shared NoC links or memory controller ports, causing queuing delays.
• Physical Layout: Cores farther from the memory controller/physical I/O (PHY) experience higher base latency. Architects use techniques like bank interleaving and multiple memory channels to distribute traffic and reduce average distance. This intra-chip latency is a major focus for NPU designers aiming for single-digit nanosecond access to on-chip SRAM or scratchpad memory.

In systems with virtual memory, every memory access by a program uses a virtual address. Translating this to a physical address adds latency before the actual memory access can begin.

• Translation Lookaside Buffer (TLB): A small, fast cache for recent virtual-to-physical page translations. A TLB hit adds just 1-2 cycles.
• TLB Miss Penalty: A TLB miss triggers a page table walk, which may require multiple sequential memory reads (to DRAM itself) to find the translation. This can add hundreds of cycles of latency.
• Impact on NPUs: Many NPUs and accelerators use physical addresses or large pages (e.g., 1GB) to minimize TLB miss rates. Management of the memory management unit (MMU) and TLB is a key part of low-latency driver and runtime design for accelerator workloads.

In multi-core, multi-accelerator systems, ensuring memory consistency across agents introduces latency through synchronization protocols.

• Cache Coherence Traffic: In a coherent system (e.g., CPU+NPU), a read may need to invalidate or fetch the latest copy of data from another agent's cache, adding network round-trips before the local read can complete.
• Memory Barriers/Fences: Instructions like __sync_synchronize() or std::atomic_thread_fence force completion of all prior memory operations before proceeding. They often flush write buffers and wait for acknowledgements, directly increasing observed latency for the subsequent operation.
• Atomic Operation Latency: Atomic operations (e.g., compare-and-swap) require read-modify-write cycles with guaranteed exclusivity, which often involves locking a cache line or memory location, resulting in latencies significantly higher than simple reads or writes. Minimizing cross-device synchronization is crucial for NPU performance.

Caching is the foundational technique for mitigating latency by storing copies of frequently accessed data in small, fast memory structures closer to the processor. It exploits temporal locality (recently used data is likely to be used again) and spatial locality (data near recently used data is likely to be used). Modern systems employ multi-level caches (L1, L2, L3). A cache hit provides data in a few cycles, while a cache miss triggers a costly access to lower memory levels. Effective cache utilization is critical for NPU performance, as neural network weights and activations often exhibit high locality.

Prefetching is a proactive technique where hardware or software predicts future memory accesses and loads data into a cache before it is explicitly requested by the processor. This aims to hide latency by overlapping data movement with computation.

• Hardware Prefetching: Dedicated units detect sequential or strided access patterns (common in array processing) and issue speculative fetches.
• Software Prefetching: Compiler or programmer inserts explicit prefetch instructions (e.g., __builtin_prefetch in C) to bring data into cache lines ahead of time. For NPUs, intelligent prefetching of weight tiles and activation blocks is essential to keep computational units fed and avoid pipeline stalls.

Scratchpad Memory (SPM) is a small, software-managed on-chip SRAM common in NPUs and DSPs. Unlike hardware-managed caches, the compiler or programmer explicitly controls data movement into and out of the SPM. This provides deterministic, low-latency access and eliminates cache management overhead (tags, miss handling). It is ideal for staging predictable data blocks like convolutional kernel weights or intermediate tensor results. Effective use requires careful memory orchestration to overlap computation with DMA transfers between global DRAM and the SPM.

This software technique reorganizes data in memory to match the expected memory access pattern of the computational kernel, maximizing cache line utilization and enabling effective prefetching.

• Tiling (Blocking): Large matrices or tensors are decomposed into smaller sub-blocks (tiles) that fit into cache or SPM. Computation proceeds on a tile-by-tile basis, dramatically improving data reuse and locality.
• Layout Transformation: Converting data from a row-major to a column-major format (or to a custom, hardware-optimized layout like NHWC for convolutions) ensures that sequential memory accesses by the processor correspond to sequential addresses, minimizing cache misses. This is a core optimization in NPU compilers for neural networks.

Memory Access Coalescing is a critical optimization for parallel architectures like NPUs and GPUs. It combines multiple simultaneous memory requests from different threads (or processing elements) into a single, wider transaction. For example, if 32 threads each request a 4-byte float value from consecutive addresses, a coalesced unit can service all requests with one 128-byte cache line fetch. Uncoalesced, scattered accesses result in many small, inefficient transactions. Achieving coalescing requires careful thread scheduling and data organization to ensure that concurrent accesses form contiguous, aligned blocks.

Direct Memory Access (DMA) allows an accelerator (NPU) or I/O device to transfer large blocks of data directly to/from host or device memory without continuous CPU intervention. This offloads the CPU and enables asynchronous data transfer.

• The CPU or NPU controller sets up a DMA descriptor (source, destination, size).
• The DMA engine executes the transfer in the background while computation proceeds on previously loaded data.
• This effectively hides the latency of data movement by overlapping it with computation, a principle known as double-buffering or ping-pong buffering, where one buffer is computed on while the other is being filled via DMA.

Cache coherence is a property of a multi-processor system that ensures all processor caches have a consistent view of shared memory. It prevents different cores or processors from reading stale or conflicting values for the same memory location, which is critical for correct parallel execution.

• Protocols: Implemented via hardware protocols like MESI (Modified, Exclusive, Shared, Invalid).
• Overhead: Maintaining coherence generates additional traffic on the memory/interconnect network, which can impact effective latency and bandwidth.
• NPU Context: In systems with NPUs and host CPUs, coherence protocols manage data visibility between the accelerator's local memory and the host's system memory.

Memory bandwidth is the maximum rate at which data can be read from or written to a memory system, measured in bytes per second (e.g., GB/s). While latency is the time to access the first byte, bandwidth is the rate of sustained data transfer.

• Relationship to Latency: High bandwidth does not guarantee low latency. A system can have high throughput for large, sequential accesses (good bandwidth) but still suffer from long initial access times (poor latency).
• Bottleneck: Compute-intensive workloads on NPUs are often bandwidth-bound, meaning the processor's computational units are stalled waiting for data to be delivered, even if latency for individual accesses is optimized.

These are two fundamental principles of program behavior that predict memory access patterns and are exploited by cache hierarchies to reduce effective latency.

• Temporal Locality: The tendency that a memory location, once accessed, is likely to be accessed again soon. Exploited by keeping recently used data in fast caches.
• Spatial Locality: The tendency that if a memory location is accessed, nearby memory locations are also likely to be accessed soon. Exploited by fetching data in blocks (cache lines).
• NPU Optimization: Kernel and data layout optimizations for NPUs explicitly engineer these localities—e.g., tiling operations to fit a matrix tile in fast scratchpad memory (temporal) and accessing elements in sequential order (spatial).

Prefetching is a hardware or software technique that predicts future memory accesses and proactively loads data into a cache or buffer before the processor explicitly requests it. Its goal is to hide memory latency by overlapping data movement with computation.

• Hardware Prefetching: Done autonomously by the memory controller or cache unit based on detected access patterns (e.g., stride, sequential).
• Software Prefetching: Explicit instructions (like prefetch) inserted by the programmer or compiler to hint at future needs.
• NPU Use: Critical for feeding NPU compute pipelines. Compilers for accelerators often insert sophisticated prefetch instructions to stream weights and activations from external DRAM (e.g., HBM) into on-chip buffers ahead of their use.

Scratchpad memory (SPM) is a small, software-managed, on-chip memory in accelerators like NPUs. Unlike a hardware-managed cache, the compiler explicitly controls what data is placed in the SPM and when, offering predictable, ultra-low-latency access.

• vs. Cache: Eliminates the latency and power overhead of cache lookup logic, tag checks, and coherence protocols. Performance is deterministic.
• Management: The programmer/compiler is responsible for data movement between SPM and larger, slower memory (DMA).
• Application: Essential for NPU performance. Critical data (e.g., a tile of a matrix multiplication) is staged in SPM for repeated, fast access, directly reducing the latency seen by the compute units.

Non-Uniform Memory Access (NUMA) is a multiprocessor memory architecture where access time to shared memory depends on the memory location's physical distance from the requesting processor. A processor has lower latency to its local memory than to memory attached to another processor.

• Latency Implication: Introduces a spectrum of access latencies within a single system, complicating performance optimization.
• System Design: Common in modern multi-socket servers. Workload and data placement must be "NUMA-aware" to minimize remote memory accesses.
• NPU Systems: In a server with multiple NPUs, a similar topology exists. An NPU may have fast access to its attached High Bandwidth Memory (HBM) but higher latency to another NPU's HBM or host CPU memory, forming a NUMA-like hierarchy for accelerators.