Memory Coalescing

Memory coalescing occurs when consecutive threads in a parallel execution unit (like a warp or wavefront) access consecutive memory addresses. For example, if thread 0 reads address A, thread 1 reads A+4, thread 2 reads A+8, and so on, the hardware can combine these into one wide load instruction. This is the ideal coalesced access pattern. The opposite, a scattered access pattern, where threads read from random, non-sequential addresses, forces multiple small transactions, wasting bandwidth and increasing latency.

Modern NPU/GPU memory controllers are optimized for large, aligned transfers. When a coalesced request is detected, the hardware issues a single transaction for a large, contiguous block of memory (e.g., 32, 64, or 128 bytes) that encompasses all requested addresses. This single transaction has nearly the same latency as a small, uncoalesced one but delivers vastly more data. The key metrics here are transaction size and memory alignment. Misaligned accesses can force two transactions instead of one, even if the pattern is contiguous.

Memory coalescing is critical for memory-bound workloads, where performance is limited by the speed of data movement, not computation. Poor coalescing directly reduces effective memory bandwidth, the usable fraction of the hardware's peak theoretical bandwidth.

• Example: A kernel with 50% coalescing efficiency on a 1 TB/s memory system behaves as if it only has 500 GB/s of bandwidth.
• This bottleneck causes compute units to stall, idling while waiting for data, which is visible in profilers as low ALU utilization or high memory pipeline occupancy.

Achieving coalescing requires designing data structures for parallel access. The two primary strategies are:

• Array of Structures (AoS): struct {float x, y, z;} points[N]; If threads access points[threadIdx].x, accesses are coalesced. If they access points[0..N].x (a strided access), they are not.
• Structure of Arrays (SoA): struct {float x[N], y[N], z[N];} points; Accessing points.x[threadIdx] is always coalesced, as all threads read contiguous elements from the x array. SoA is generally preferred for NPU/GPU programming to guarantee coalesced access across different attributes.

Memory coalescing is a primary target for auto-tuning systems. Kernel tuners experiment with parameters like thread block size and memory tile dimensions to find configurations that promote coalesced access. Profiling tools provide direct feedback:

• Performance counters for global load/store efficiency measure the ratio of coalesced transactions to total requests.
• Execution traces can visualize memory access patterns to identify scattered reads/writes.
• Bottleneck analysis will flag memory coalescing issues as a top cause of memory-bound behavior, guiding optimization efforts.

While coalescing optimizes main memory (DRAM) access, it also dramatically improves cache hit rates. A single, wide coalesced load efficiently fills an entire cache line. Subsequent accesses by threads in the same group to data within that cache line are served from fast on-chip memory (L1/L2 cache). Poorly coalesced accesses waste cache space by bringing in small, disjoint pieces of data, leading to cache thrashing and reduced effective bandwidth from the cache subsystem as well.

The maximum rate at which data can be read from or written to a memory subsystem by a processor, measured in gigabytes per second (GB/s). Memory coalescing directly maximizes the effective utilization of this bandwidth by reducing the number of required transactions. A workload is memory-bound when its performance is limited by this bandwidth, making coalescing a critical optimization.

A condition in parallel computing where threads within the same warp or wavefront follow different execution paths due to conditional branches (e.g., if/else statements). This causes serialization, as all paths must be executed sequentially, drastically reducing efficiency. Poor memory access patterns often correlate with control flow divergence, both of which degrade SIMD (Single Instruction, Multiple Data) utilization on NPUs and GPUs.

The percentage of memory access requests successfully served from a fast cache memory, versus requiring a slower access to main memory (DRAM). While coalescing optimizes DRAM access, complementary techniques like tiling are used to improve cache locality. A high cache hit rate reduces pressure on the memory controller and works synergistically with coalesced DRAM accesses to minimize latency.

The number of data elements processed simultaneously by a Single Instruction, Multiple Data (SIMD) or vector instruction unit. On modern NPUs, memory load/store instructions are inherently vectorized. Memory coalescing ensures that the data fetched by a vector load instruction is contiguous and useful to all threads, preventing wasted bandwidth. The optimal vectorization factor is often a tunable parameter in auto-tuning.

The number of threads grouped together for cooperative execution and synchronization within a single compute unit. This is a critical tunable parameter in auto-tuning. The chosen size must align with the hardware's execution width (warp/wavefront size) and memory transaction granularity to enable effective memory coalescing. A misaligned workgroup size can striddle transaction boundaries and degrade performance.

The automated process of searching a configuration space (e.g., workgroup size, tile dimensions, loop unroll factor) to find the optimal parameters for a computational kernel on specific hardware. Tools called kernel tuners often evaluate configurations based on performance models or direct measurement. Memory coalescing is a primary optimization objective that guides the search for optimal memory access patterns.