Memory Bandwidth

The fundamental hardware specification. Bandwidth (GB/s) = (Bus Width in bits × Data Rate per pin in GT/s) / 8. The data rate is often a multiple of the base clock frequency (e.g., DDR5-6400 operates at 6400 MT/s). Wider buses and higher frequencies directly increase theoretical peak bandwidth. NPUs often utilize High Bandwidth Memory (HBM) with very wide interfaces (1024-bit to 4096-bit) to achieve extreme bandwidths exceeding 1 TB/s.

How software organizes memory requests. Memory coalescing is the critical optimization where consecutive threads access consecutive memory addresses, allowing the memory controller to service them with a single, wide transaction. Scattered, random accesses force many small transactions, wasting bandwidth. Profilers measure memory transaction efficiency to identify poor coalescing. Optimizing data layouts (Structure of Arrays vs. Array of Structures) is a primary technique to improve coalescing.

The effectiveness of on-chip caches (L1, L2, shared memory) at reducing demands on external DRAM. A high cache hit rate means most data is served from fast, on-chip memory, reducing traffic on the high-latency, bandwidth-constrained path to main memory. Inefficient reuse of data (poor temporal locality) or accessing large working sets that don't fit in cache (poor spatial locality) leads to cache thrashing and saturates the main memory bandwidth.

Maximizing parallel access to memory resources. DRAM and on-chip memories (like shared memory on NPUs) are partitioned into banks. Simultaneous accesses to different banks can proceed in parallel. Bank conflicts occur when multiple threads attempt to access the same bank simultaneously, causing serialization and reducing effective bandwidth. Auto-tuners often search for optimal memory stride and workgroup sizes to minimize bank conflicts.

Hardware and software mechanisms to anticipate data needs. Modern memory controllers perform automatic prefetching, detecting sequential access patterns and loading subsequent data into buffers before it's explicitly requested. Software can use prefetch intrinsics to hint at future accesses. Memory subsystems are optimized for burst transfers; a single access to open a DRAM row is expensive, but subsequent reads from that row are cheap. Algorithms should be designed for long, contiguous bursts.

The contention point for all memory requests. The memory controller has finite queues and scheduling logic. When too many concurrent kernels, DMA engines, or host CPU requests flood the controller, it becomes saturated. This leads to increased arbitration overhead and can prevent the theoretical bus bandwidth from being achieved. Profiling memory controller utilization and managing concurrent kernel execution are necessary to avoid this bottleneck.

A fundamental optimization where consecutive threads access consecutive memory addresses, enabling a single, wide memory transaction. Non-coalesced access forces multiple smaller transactions, wasting bandwidth.

• Goal: Minimize the number of memory transactions per warp/wavefront.
• Implementation: Structure data layouts (e.g., Structure of Arrays) and kernel indexing to ensure aligned, sequential access patterns.
• Impact: Can improve effective bandwidth by 10x or more on modern NPU/GPU architectures.

A technique that uses fast, programmer-managed on-chip SRAM (shared memory/L1 cache) as an explicit user-managed cache to reduce accesses to slower global memory.

• Process: Load a tile of data from global memory into shared memory. Threads within a workgroup collaboratively reuse this tile.
• Benefit: Dramatically reduces global memory bandwidth demands for algorithms with data reuse (e.g., matrix multiplication, convolutions).
• Key Tuning Parameter: Tile size, balanced between shared memory capacity and occupancy.

The proactive loading of data into a cache or register before it is explicitly needed by the compute units, hiding memory access latency.

• Software Prefetching: Explicit compiler intrinsics or instructions that hint to the memory controller to load specific data.
• Hardware Prefetching: Automatic mechanisms in the memory controller that detect sequential access patterns.
• Use Case: Crucial for irregular access patterns where hardware prefetchers fail. Effective prefetching can keep the compute pipeline full.

Designing data structures and traversal algorithms to maximize spatial and temporal locality.

• Spatial Locality: Accessing data elements that are stored close together in memory (exploits cache lines).
• Temporal Locality: Reusing the same data elements within a short time period (keeps data in cache).
• Common Techniques: Loop tiling, loop interchange, and using SoA (Structure of Arrays) over AoS (Array of Structures) for SIMD architectures.

A compiler-level optimization that merges multiple computational kernels into a single kernel, eliminating intermediate writes and reads to global memory.

• Bandwidth Benefit: The fused kernel passes intermediate results directly through registers or shared memory, bypassing costly round-trips to DRAM.
• Example: Fusing a convolution, activation function, and pooling layer into one kernel.
• Trade-off: Increases register pressure and kernel complexity, potentially reducing occupancy.

Using non-blocking memory transfers and streams to overlap data movement (Host-to-Device, Device-to-Host) with kernel execution on the NPU.

• Mechanism: While one kernel is processing data in memory A, data for the next kernel can be asynchronously loaded into memory B.
• Tools: APIs like cudaMemcpyAsync (NVIDIA) or equivalent in vendor SDKs.
• Outcome: Effectively hides PCIe or internal bus latency, making memory bandwidth less of a serial bottleneck for pipelined workloads.

A computational state where a kernel's execution time is limited by the speed or bandwidth of the memory subsystem, not the processor's arithmetic capabilities. On an NPU, this occurs when compute units spend significant time idle, waiting for data to be fetched from DRAM or lower-level caches.

Key Indicators:

• High cache miss rate and memory controller utilization.
• Low compute throughput (FLOPS/TOPS) relative to peak.
• Performance scales directly with increased memory clock speed or bus width.

Optimization Focus: Techniques like memory coalescing, improved data locality, and using faster memory (HBM) are critical to alleviate this bottleneck.

The rate at which a processor completes computational operations, measured in operations per second (e.g., FLOPS for floating-point, TOPS for integer operations). For NPUs, this is the peak theoretical performance when not limited by memory bandwidth.

Relationship to Bandwidth: A kernel is memory-bound when its arithmetic intensity (operations per byte of data moved) is too low to saturate the available compute throughput. The Roofline Model visually plots this relationship, showing the performance limit imposed by either peak compute (the "roof") or available bandwidth (the "slope").

A critical memory access pattern optimization for parallel architectures like NPUs and GPUs. It occurs when consecutive threads in a warp or wavefront access consecutive memory addresses, allowing the memory controller to combine these requests into a single, wide transaction.

Impact on Bandwidth:

• Coalesced access: Maximizes effective bandwidth by utilizing the full width of the memory bus.
• Uncoalesced access: Causes multiple smaller transactions, drastically reducing effective bandwidth and increasing latency.

Implementation: Requires structuring data layouts (e.g., Structure of Arrays) and thread indexing to ensure adjacent threads read adjacent data elements.

The percentage of memory access requests that are successfully served from a fast, on-chip cache (e.g., L1, L2, shared memory) instead of requiring a slower access to main DRAM. A high cache hit rate is essential for reducing effective memory latency and pressure on the external memory bandwidth.

For NPU Performance:

• High Hit Rate: Data is reused locally, minimizing off-chip bandwidth demands.
• Low Hit Rate: Constant DRAM fetches, exposing the full latency and bandwidth limits of the memory subsystem.

Optimization Techniques: Loop tiling (tile size selection), data prefetching, and exploiting memory hierarchy (global -> shared -> register) are used to improve locality and hit rates.

A kernel's ratio of arithmetic operations (FLOPs) to bytes of data transferred between the processor and main memory. It is a fundamental metric for determining if a workload is compute-bound or memory-bound.

Formula: AI = (Total FLOPs) / (Total Bytes Transferred)

Interpretation:

• Low AI (< ~10 FLOPs/byte): Typically memory-bound. Performance is limited by bandwidth.
• High AI (> ~100 FLOPs/byte): Typically compute-bound. Performance is limited by peak FLOPS.

Use in Roofline Analysis: Plotted on the x-axis, it determines which performance ceiling (compute or memory) applies to a given kernel on specific hardware.

The systematic process of identifying the primary limiting factor that restricts the overall performance of a system or application. For NPU workloads, the core dichotomy is between compute bottlenecks and memory bottlenecks.

Methodology:

1. Profile using kernel profilers and performance counters.
2. Measure key metrics: achieved FLOPS (vs. peak), memory bandwidth utilization, cache hit rates, and occupancy.
3. Analyze using models like the Roofline Model.
4. Identify the dominant bottleneck (e.g., DRAM bandwidth, L2 cache bandwidth, ALU throughput).

Outcome: Directs optimization efforts. If memory-bandwidth-bound, focus shifts to data layout, coalescing, and reducing redundant transfers.