Memory-Bound Optimization

The defining mathematical signature of a memory-bound operation is a low ratio of floating-point operations (FLOPs) to bytes of data moved. Arithmetic intensity is measured in FLOPs/Byte. Operations like element-wise addition, ReLU activation, and batch normalization perform very few calculations per data element read, making them classic memory-bound kernels. In contrast, a large matrix multiplication has high arithmetic intensity and is typically compute-bound.

A direct symptom is that the processor's memory bandwidth is saturated while its compute units remain underutilized. Profiling tools will show high DRAM read/write throughput near the hardware's theoretical peak, but low core utilization or streaming multiprocessor (SM) occupancy. The processor is constantly stalled, waiting for data to arrive from main memory or the last-level cache.

Memory-bound workloads are prime candidates for operator fusion. This compiler optimization combines a sequence of low-arithmetic-intensity operations (e.g., a convolution followed by a bias add and a ReLU) into a single kernel. By doing this, intermediate results are kept in fast on-chip registers or shared memory instead of being written back to slow global memory, dramatically reducing the total number of memory transactions.

Performance is highly sensitive to how tensors are organized in memory. Accessing data with a non-coalesced stride (e.g., reading a column from a row-major matrix) causes cache line thrashing and poor bandwidth utilization. A key optimization is transforming data from a channels-first (NCHW) to a channels-last (NHWC) layout, or vice-versa, to ensure that the innermost loop accesses contiguous memory addresses, maximizing cache line usage.

A primary algorithmic fix for memory-bound kernels is cache blocking, also known as tiling. The working set of data is partitioned into smaller blocks that fit entirely within a faster level of the memory hierarchy (e.g., L2 cache or shared memory). The algorithm then iterates over these blocks, reusing data extensively before moving to the next block. This maximizes temporal locality and minimizes expensive trips to main memory.

A loop transformation technique that partitions data into smaller blocks (tiles) that fit into the processor's cache hierarchy. By reusing data while it resides in faster, low-latency memory (L1/L2 cache), cache blocking drastically reduces expensive trips to main memory (DRAM).

• Goal: Maximize temporal data locality.
• Mechanism: Reorder nested loops to operate on a sub-matrix or tile before moving to the next.
• Impact: Critical for matrix multiplication and convolutions where naive implementations constantly evict data from cache.

The process of changing how multi-dimensional tensors are stored in linear memory to improve access patterns. The optimal layout depends on the compute kernel's access stride.

• NCHW vs. NHWC: Channel-first (NCHW) often favors convolutions, while channel-last (NHWC) can be faster for activation functions on specific hardware.
• Struct of Arrays (SoA) vs. Array of Structs (AoS): SoA layout enables coalesced memory access for vectorized operations, preventing cache line waste.
• Goal: Ensure the hardware prefetcher can predict access patterns and that SIMD units load contiguous data.

An explicit instruction to the processor to load data into the cache before it is actually needed by the computation. This hides memory latency by overlapping data movement with arithmetic operations.

• Mechanism: Compiler intrinsics (e.g., __builtin_prefetch) or hardware auto-prefetchers.
• Temporal vs. Spatial: Prefetch instructions can specify which cache level to load into and whether to evict data immediately after use.
• Risk: Over-aggressive prefetching can pollute the cache with unused data, evicting useful data and degrading performance.

A measure of how effectively a workload uses the available data throughput between the processor and memory. The Roofline Model plots operational intensity (FLOPs/byte) against peak compute and memory bandwidth to identify if a kernel is memory-bound.

• Arithmetic Intensity: The ratio of compute operations to bytes transferred. Low intensity implies a memory-bound kernel.
• Optimization: Fusing layers (operator fusion) increases arithmetic intensity by performing multiple calculations on a single data load, reducing total bytes transferred.

In multi-socket systems, memory access latency varies depending on the physical location of the data relative to the core. NUMA-aware allocation binds threads and their memory pages to the same physical socket.

• First-Touch Policy: The operating system allocates memory on the node where the thread that first writes to the page runs.
• numactl: A Linux utility to bind processes and memory to specific nodes.
• Impact: Prevents remote memory access penalties that can halve effective bandwidth in data-parallel inference workloads.

Reducing the size of model weights in memory to lower the volume of data that must be fetched during inference. While often discussed for storage reduction, low-bit quantization directly addresses memory bandwidth bottlenecks.

• INT4/INT8 Inference: Loading a 4-bit weight requires 1/4th the bandwidth of a FP16 weight.
• Memory-Bound Gains: In large language model decoding, the primary bottleneck is reading the weight matrix from VRAM. 4-bit quantization can nearly double token generation speed by halving the data transfer time.