Memory Bandwidth

Theoretical peak bandwidth is the maximum possible data rate calculated from hardware specs (bus width × clock speed). Effective bandwidth is the actual, sustained rate achieved by a real workload, which is always lower. The gap is caused by:

• Protocol overhead from command/address cycles.
• Inefficient access patterns that don't saturate the bus.
• Bank conflicts and row buffer misses in DRAM.
• Arbitration delays in shared memory controllers. For NPU performance modeling, effective bandwidth is the critical metric.

These are distinct, often opposing, performance dimensions. Latency is the time to fetch the first byte of data. Bandwidth is the rate for transferring a large block of data after the initial fetch.

• A high-bandwidth system (e.g., using HBM) may still suffer if workloads have poor locality, causing frequent high-latency accesses.
• Optimizing for one can hurt the other; wide burst transfers maximize bandwidth but increase latency if the requested data isn't contiguous. NPU kernels must be designed to amortize latency over large, contiguous transfers to fully utilize available bandwidth.

Modern memory interfaces (e.g., GDDR6, HBM2e) are optimized for long, contiguous burst transfers. Efficiency requires:

• Memory alignment of data addresses to match burst boundaries.
• Coalesced memory accesses where multiple parallel threads access contiguous words, combining them into a single wide transaction.
• Prefetching to stream data ahead of the compute units' needs. Poorly structured kernels cause short, unaligned transfers, leaving the data bus underutilized and drastically reducing effective bandwidth.

Memory bandwidth is not always symmetric for read and write operations. This is due to hardware optimizations:

• Write buffers can allow writes to be acknowledged quickly, but the actual data write to memory may be slower or consume less immediate bandwidth.
• Reads are typically latency-critical and stall the processor, making read bandwidth paramount for performance.
• Some NPU architectures have separate read/write paths with different widths or clocking. Workload analysis must consider the bandwidth ratio of weights/input reads (high) versus activation writes (lower).

Bandwidth is a shared resource. Contention arises from:

• Multiple cores/threads within an NPU accessing memory simultaneously.
• DMA engines performing background data transfers.
• The host CPU accessing the same memory pool in a unified memory architecture. Contention leads to non-deterministic performance. Techniques to manage it include:
• Memory partitioning and bank coloring.
• Scheduling compute and DMA transfers to avoid peaks.
• Using scratchpad memory to reduce off-chip traffic.

Achieving peak bandwidth requires massive parallelism to issue enough concurrent memory requests. This is enabled by:

• Wide SIMD/SIMT units in NPUs that generate hundreds of concurrent memory requests.
• Multiple memory controllers and independent channels (e.g., 1024-bit HBM interfaces).
• Deep memory request queues to hide latency and keep the bus saturated. The roofline model visualizes this: performance is either compute-bound (low operational intensity) or memory-bound (high intensity), dictated by the available bandwidth.

Memory latency is the time delay between a processor's request for data and the moment the data is available for use, measured in clock cycles or nanoseconds. While bandwidth measures data volume per second, latency measures the time to fetch the first byte.

• Critical Relationship: High bandwidth is ineffective if latency is excessive, as the processor stalls waiting for the initial data.
• Hierarchy Impact: Lower levels of the memory hierarchy (e.g., L1 cache) have ultra-low latency but limited capacity, while higher levels (e.g., DRAM) have higher latency but greater capacity and bandwidth.
• NPU Consideration: NPU kernels must be designed for data reuse to amortize high DRAM latency over many computations, making effective use of on-chip scratchpad or cache memory.

High Bandwidth Memory (HBM) is a 3D-stacked DRAM technology designed specifically to provide extreme memory bandwidth in a compact form factor. It is a primary enabler for the high bandwidth requirements of modern NPUs and GPUs.

• Architecture: Multiple DRAM dies are stacked vertically and connected using Through-Silicon Vias (TSVs), creating a wide, ultra-fast interface to the processor.
• Bandwidth Advantage: HBM provides bandwidth an order of magnitude higher than traditional GDDR memory (e.g., >1 TB/s vs. hundreds of GB/s) by utilizing a very wide data bus (1024-bit or 2048-bit).
• Use Case: Essential for feeding data-hungry tensor cores in AI accelerators, where model weights and activations must be streamed at immense rates to avoid compute starvation.

A memory access pattern describes the sequence, stride, and predictability with which a program reads from or writes to memory. It is the primary determinant of achievable effective bandwidth, as hardware prefetchers and cache policies are optimized for specific patterns.

• Optimal Patterns: Sequential/contiguous access enables maximum bandwidth utilization by allowing efficient burst transfers and hardware prefetching.
• Problematic Patterns: Random access or large, non-unit strides often result in poor cache utilization and low effective bandwidth, as each access may incur a full latency penalty.
• NPU Optimization: Compilers for NPUs perform loop tiling and data layout transformations (e.g., NHWC to NCHW) specifically to create regular, coalesced access patterns that maximize bandwidth to HBM or GDDR.

Spatial locality is a principle of program behavior stating that if a particular memory location is accessed, nearby memory locations are likely to be accessed soon. Exploiting spatial locality is fundamental to achieving high memory bandwidth efficiency.

• Cache Line Basis: Modern memory systems transfer data in fixed-size blocks (cache lines, typically 64-128 bytes). Good spatial locality means each transferred block is fully utilized, maximizing bandwidth efficiency.
• Bandwidth Waste: Poor spatial locality results in low bandwidth utilization, as only a small portion of each fetched cache line is used before it is evicted.
• Design Implication: NPU data structures (e.g., tensors) and computation loops should be arranged to ensure that consecutively accessed elements are adjacent in memory.

Prefetching is a hardware or software technique that predicts future memory accesses and proactively loads data into a cache or buffer before it is explicitly requested by the processor. Its goal is to hide memory latency and keep the bandwidth pipeline full.

• Hardware Prefetching: Common in CPUs and some NPUs, it detects sequential or strided access patterns and issues speculative read requests to memory.
• Software Prefetching: Explicit instructions (prefetch intrinsics) are inserted by the programmer or compiler to hint at future data needs, providing more control for irregular patterns.
• Bandwidth Impact: Effective prefetching transforms a latency-bound workload into a bandwidth-bound one, as the processor always has data ready to consume, maximizing sustained bandwidth.

Compute Express Link (CXL) is an open industry-standard interconnect that provides high-bandwidth, cache-coherent connectivity between the host CPU and devices like accelerators (NPUs), memory expanders, and smart NICs. It is reshaping memory bandwidth architecture in heterogeneous systems.

• Memory Semantics: CXL allows an NPU to access host memory (and vice versa) using load/store instructions with low latency, creating a unified memory space. This reduces the need for explicit, bandwidth-consuming copy operations.

• Memory Pooling: CXL Type 3 devices enable memory expansion and pooling, allowing multiple hosts or accelerators to share a pool of high-bandwidth memory, dynamically allocating bandwidth and capacity as needed.

• Future Systems: CXL is key to overcoming the memory wall in data centers, enabling composable systems where NPUs can directly and efficiently access vast, shared memory resources.