Memory-Bound

Arithmetic Intensity is the ratio of floating-point operations (FLOPs) to bytes of memory accessed. Memory-bound workloads have low arithmetic intensity, meaning they perform few calculations for each byte loaded from memory. This keeps the processor's computational units idle while waiting for data.

• Example: A simple element-wise operation on a large vector (e.g., adding a constant) has an intensity of ~0.1-0.2 FLOPs/byte.
• Contrast: A dense matrix multiplication is compute-bound, with intensity often exceeding 10-100 FLOPs/byte, fully utilizing the processor's FLOP capacity.

These workloads saturate the available memory bandwidth—the maximum rate data can be read from or written to memory. Performance scales directly with bandwidth, not processor clock speed or core count.

• Key Metric: Achieved bandwidth as a percentage of the hardware's peak theoretical bandwidth (e.g., 80-95% utilization).
• Bottleneck: The memory bus becomes the limiting resource. Upgrading to a CPU/GPU with higher bandwidth (e.g., HBM2e vs. GDDR6) yields direct performance gains, while a faster compute chip does not.

Memory-bound workloads exhibit poor cache locality. Data is not reused before being evicted from the small, fast CPU caches (L1/L2/L3), leading to frequent, expensive trips to main memory (DRAM).

• Symptoms: High last-level cache miss rates observed via performance counters.
• Cause: Large working sets that exceed cache capacity or irregular, non-sequential access patterns (e.g., sparse embeddings, graph traversals). Each miss stalls the processor for hundreds of cycles.

The majority of the end-to-end inference time is spent moving model weights, activations, and key-value caches, not on computation. Time per output token (TPOT) is largely dictated by memory access latency.

• Transformer Inference: Loading the KV Cache for each layer and token can dominate latency in autoregressive decoding, especially for long contexts.
• Impact: Optimizations focus on memory layout (chunking, prefetching), compression (quantization), and caching strategies rather than parallelizing computations.

Throwing more parallel compute cores at a memory-bound problem yields diminishing returns. Performance does not scale linearly because all cores compete for the same, saturated memory bandwidth.

• Example: Running a memory-bound kernel on 32 GPU Streaming Multiprocessors (SMs) may offer only a 2-4x speedup over running it on 8 SMs.
• Solution: Optimization requires reducing the total bytes moved (working set size) via techniques like operator fusion (to keep data in registers) or using more compact data types (INT8 quantization).

The Roofline Model is the definitive analytical tool for diagnosing memory-bound performance. It plots attainable GFLOPs/sec against operational intensity.

• Plot Interpretation: A kernel whose performance point falls on the memory-bound slope of the roofline is limited by bandwidth. Its performance = (Operational Intensity) × (Peak Memory Bandwidth).
• Actionable Insight: If the kernel is on the memory-bound slope, optimize data access patterns, reduce data movement, or apply compression. If it's below the roofline, improve core utilization (compute-bound optimization).

The primary goal is to maximize data locality to reduce cache misses and improve memory bandwidth utilization. This involves restructuring computations to access data in contiguous blocks that fit within CPU/GPU cache hierarchies (L1, L2, L3). Techniques include:

• Loop tiling/blocking: Decomposing large loops into smaller blocks that operate on data subsets that remain in cache.
• Data layout transformation: Converting data structures from Array of Structures (AoS) to Structure of Arrays (SoA) to enable efficient vectorized loads.
• Prefetching: Explicitly loading data into cache before it is needed by the computational kernel, hiding memory latency.

This technique combines multiple sequential computational operations (layers or operators) into a single, fused kernel. This eliminates intermediate results that would be written to and read from main memory (DRAM). For example, fusing a GeLU activation function with the preceding linear layer avoids storing the full intermediate tensor. Benefits include:

• Reduced memory footprint: Lower peak memory consumption during inference.
• Fewer memory transactions: Decreased pressure on the memory bus.
• Improved cache efficiency: Fused data stays in registers or cache. This is a core optimization in compilers like Apache TVM, TensorRT, and XLA.

Quantization reduces the numerical precision of model weights and activations (e.g., from 32-bit floating-point FP32 to 8-bit integer INT8). This directly addresses memory-boundedness by:

• Halving memory bandwidth requirements: Loading INT8 weights uses 75% less bandwidth than FP32.
• Increasing cache effectiveness: More parameters can fit into the same cache size.
• Enabling faster integer compute units on modern hardware. Post-training quantization (PTQ) is commonly applied to memory-bound models, while quantization-aware training (QAT) can recover higher accuracy. This is critical for deploying large models like LLaMA or BERT under strict latency SLOs.

Pruning removes redundant or less important parameters from a neural network, creating a sparse model. Sparsity reduces the amount of data that must be loaded from memory. Key approaches:

• Structured Pruning: Removes entire channels, filters, or layers, leading to immediate memory and FLOP reduction.
• Unstructured Pruning: Sets individual weights to zero. To realize memory bandwidth gains, specialized sparse tensor formats (e.g., CSR, CSC) and hardware with sparse compute support (like NVIDIA's Sparse Tensor Cores) are required to skip loading zero values. Sparse models can reduce memory traffic by 30-50% for transformer-based architectures.

Dynamic memory allocation during inference (e.g., for temporary tensors) introduces overhead and fragmentation. Optimization strategies include:

• Static memory planning: Pre-allocating a single, large contiguous buffer at model load time and sub-allocating all intermediate tensors from it. This eliminates runtime malloc/free calls.
• Memory reuse: Identifying non-overlapping lifetimes of tensors and assigning them to the same memory region.
• GPU-specific optimizations: Using cudaMallocAsync with a memory pool to reduce allocation latency and fragmentation on NVIDIA GPUs. Frameworks like TensorFlow and PyTorch employ memory allocators with these features to minimize memory-bound overhead.

The Roofline Model is an essential analytical tool for diagnosing memory-bound workloads. It plots attainable performance (FLOPs/sec) against operational intensity (FLOPs/byte).

• Operational Intensity: Calculated as the total floating-point operations divided by the total bytes moved between memory and the processor.
• Performance Ceilings: The plot shows a horizontal ceiling (compute-bound) and a diagonal ceiling (memory-bound). If a kernel's performance point falls on the diagonal slope, the workload is memory-bound. The optimization goal is to increase operational intensity (via techniques like fusion or tiling) to move the point toward the horizontal, compute-bound ceiling.

A compute-bound workload is limited by the available computational capacity (FLOPs) of the processor rather than by memory bandwidth or I/O. This is the opposite of a memory-bound state.

• Key Characteristic: High arithmetic intensity (many operations per byte of data loaded).
• Typical Scenario: Dense matrix multiplications in large linear layers, or running models with heavily quantized weights that reduce memory traffic.
• Optimization Focus: Maximizing GPU/CPU utilization, using tensor cores, and applying operator fusion to reduce kernel launch overhead.

The roofline model is an analytical performance model that visualizes the attainable performance of a computational kernel as a function of its operational intensity.

• Operational Intensity: Measured in operations per byte (Ops/Byte), it defines the balance between compute and memory access.
• The Roofline: The model plots a "roof" representing the hardware's limits: a flat, memory-bandwidth-bound ceiling at low intensity, and a sloped, compute-bound ceiling at high intensity.
• Diagnostic Use: Plots the performance of a specific kernel. If it's below the roof, it's optimization-bound; if it's on the memory ceiling, it's memory-bound; if on the compute ceiling, it's compute-bound.

Arithmetic Intensity is a key metric that determines whether a workload is memory-bound or compute-bound. It is calculated as the number of arithmetic operations performed per byte of data transferred from memory.

• Formula: AI = (Total FLOPs) / (Total Bytes Accessed from Memory).
• Low Intensity (< ~10 Ops/Byte for modern GPUs): Indicates a memory-bound workload. Common in element-wise operations, attention score calculations, and embedding lookups.
• High Intensity (> ~100 Ops/Byte): Indicates a compute-bound workload. Common in large, dense matrix multiplications.
• Bridge to Hardware: This metric must be compared to the system's machine balance (Peak FLOPs / Peak Memory Bandwidth) to identify the limiting factor.

Memory Bandwidth is the maximum rate at which data can be read from or written to a memory subsystem (e.g., GPU VRAM, CPU RAM). It is the critical constraint for memory-bound workloads.

• Measurement: Typically in GB/s or TB/s.
• Hardware Dependency: Determined by the memory technology (e.g., HBM2e, GDDR6) and bus width. It is a fixed hardware ceiling.
• Impact on Inference: In a memory-bound scenario, the actual achieved throughput is directly proportional to the available memory bandwidth and the inverse of the model's activation size per layer. Optimizations focus on reducing the total bytes transferred.

Latency is the time delay between submitting an inference request and receiving the response. Memory-bound operations are a primary contributor to latency.

• Memory Latency vs. Bandwidth: Latency is the time to access the first byte; bandwidth is the rate for the entire transfer. Memory-bound issues are usually about saturating bandwidth, not just latency.
• Effect: In memory-bound layers, the processor spends most of its time waiting for data to be fetched, increasing Time per Output Token (TPOT).
• Measurement: Critical percentiles (P99 latency) are often dominated by memory subsystem behavior under load.

Hardware Utilization measures the percentage of available computational resources actively used during execution. It provides a clear signal of a performance bottleneck.

• Low Compute Utilization with High Memory Traffic: A classic signature of a memory-bound kernel. The compute cores are idle, waiting for data.
• Tools: Profilers like NVIDIA Nsight Systems show SM (Streaming Multiprocessor) Utilization and Memory Bandwidth Usage concurrently.
• Goal: For memory-bound workloads, optimization aims not to increase compute utilization (which is already low due to stalling), but to reduce the required memory traffic, thereby allowing the same computation to finish faster.