Memory Bandwidth

Memory bandwidth is the maximum theoretical data transfer rate between a processor (CPU, GPU, NPU) and its main memory (DRAM). It is calculated as:

Bandwidth = (Memory Bus Width × Memory Clock Speed × Transfers per Clock) / 8

• Memory Bus Width: The number of data lines (e.g., 128-bit, 256-bit).
• Memory Clock Speed: The frequency of the memory interface (e.g., 3200 MHz).
• Transfers per Clock: For DDR (Double Data Rate) memory, this is 2.
• The result is expressed in bytes per second (e.g., GB/s). This rate represents a hard physical limit for how quickly model weights and activation tensors can be loaded for computation.

For most edge AI inference tasks, the system is memory-bound, not compute-bound. This is because:

• Large Model Footprints: Even compressed models (e.g., 50-100 MB) exceed on-chip SRAM cache, requiring frequent DRAM access.
• Data-Intensive Operations: Layers like fully connected and attention have low arithmetic intensity, meaning they perform few calculations per byte fetched from memory.
• Underutilized Compute: High-performance NPU/GPU tensor cores sit idle waiting for data, a state known as the memory wall. Performance is capped by the roofline model's memory-bound plateau.

Optimizing memory access patterns is therefore more critical than raw peak TOPS (Tera Operations Per Second) for real-world latency.

Accessing off-chip DRAM is one of the most power-expensive operations on an edge SoC (System-on-Chip).

• Dynamic Power: DRAM access power scales with bandwidth utilization. High, sustained bandwidth directly drains battery.
• Static Power: The memory interface and PHY (physical layer) consume power even when idle.
• Optimization Strategy: The goal is to minimize DRAM traffic. Techniques include:
  • Model compression (pruning, quantization) to reduce data size.
  • Kernel fusion to keep intermediate tensors in fast on-chip cache.
  • Data layout optimization (e.g., NHWC vs. NCHW) to enable contiguous, cache-friendly access.

Lower effective bandwidth often translates directly to longer device battery life.

Memory bandwidth influences both average latency and tail latency, which is critical for real-time systems.

• Variable Access Latency: DRAM access time depends on row buffer hits/misses, leading to non-deterministic latency. A cache miss requiring a DRAM row activation adds significant, unpredictable delay.
• Contention: In heterogeneous computing systems (CPU, NPU, GPU sharing memory), contention for bandwidth can cause jitter, violating deterministic execution guarantees.
• Mitigation: Techniques to ensure predictable bandwidth include:
  • Memory bandwidth reservation or partitioning.
  • Worst-Case Execution Time (WCET) analysis that accounts for worst-case memory access patterns.
  • Using scratchpad SRAM (software-managed cache) for time-critical data paths.

Achievable bandwidth is dictated by the physical hardware architecture:

• Memory Type: LPDDR5 offers higher bandwidth and better power efficiency than LPDDR4X for mobile/edge SoCs.
• Bus Configuration: A 128-bit bus provides half the theoretical bandwidth of a 256-bit bus at the same clock speed.
• Shared vs. Dedicated Memory: In many edge SoCs, the NPU shares the main system DRAM with the CPU, leading to contention. High-end accelerators may have dedicated, wider memory interfaces (e.g., HBM).
• Packaging: Advanced packaging like 2.5D/3D integration (e.g., using an interposer) can provide immense bandwidth by placing memory dies very close to the processor, but at higher cost.

System architects must select silicon where the memory subsystem bandwidth aligns with the target model's requirements.

Effective optimization requires measuring actual, not just theoretical, bandwidth utilization.

• Performance Counters: Use hardware performance monitoring units (PMUs) to track metrics like DRAM_READS and DRAM_WRITES.
• Roofline Analysis: Plot the operational intensity (FLOPs/byte) of model layers against the system's roofline to identify if they are compute-bound or memory-bound.
• Profiling Tools: Frameworks like NVIDIA Nsight Systems, ARM Streamline, or Intel VTune provide memory bandwidth timelines.
• Key Insight: A layer operating far below peak compute FLOPS is likely starving for data. The solution is to increase data reuse (e.g., via tiling) or reduce data movement, not to add more compute cores.

A workload is compute-bound when its execution time is limited by the speed of the processor's arithmetic units, not by the rate of data transfer to/from memory. This is the opposite of a memory bandwidth bottleneck.

• Characteristics: High operational intensity (FLOPs per byte of data).
• Edge AI Context: Deep, computationally heavy layers (like large fully connected layers) can become compute-bound, especially on powerful NPUs.
• Optimization Focus: Maximizing FLOP/s utilization through techniques like kernel fusion and efficient use of tensor cores.

The roofline model is an analytical performance model that visualizes the attainable performance of a computational kernel. It plots performance (FLOP/s) against operational intensity (FLOPs/byte).

• Two Limits: Performance is bounded by either the system's peak compute throughput (a flat ceiling) or its peak memory bandwidth (a sloping roof).
• Diagnostic Tool: Placing a kernel on the roofline chart instantly shows if it is compute-bound (near the flat ceiling) or memory-bound (on the sloping roof).
• Design Use: Guides optimization efforts—memory-bound kernels benefit from data reuse and cache optimization, while compute-bound kernels need better parallelization.

Activation sparsity is a property where many of the output values (activations) from a neural network layer are zero, often due to the use of ReLU or similar activation functions.

• Bandwidth Impact: Sparse activations mean many memory reads/writes are for zero values, which is inefficient.
• Hardware Exploitation: Specialized accelerators (e.g., some NPUs) can skip computations and compress data transfers for zero values, effectively reducing the required memory bandwidth.
• Induced Sparsity: Techniques like pruning and training with sparsity-inducing regularizers can increase activation sparsity to leverage these hardware benefits.

Kernel fusion is a compiler optimization that combines multiple sequential computational operations (kernels) into a single, larger kernel.

• Bandwidth Benefit: Fuses operations to keep intermediate data in fast on-chip memory (cache/registers), eliminating costly writes and reads to/from main memory (DRAM).
• Example: Fusing a convolution, batch normalization, and ReLU activation into one kernel prevents writing the convolution output to DRAM only to immediately read it back for the next operation.
• Result: Reduces memory traffic, alleviates bandwidth pressure, and decreases latency by reducing kernel launch overhead.

Operations per watt (often Ops/W or FLOPs/W) is a key efficiency metric for edge AI, measuring the number of computational operations a system can perform for each joule of energy consumed.

• Direct Relationship to Bandwidth: Accessing DRAM is one of the most power-expensive operations in a chip. High, inefficient memory bandwidth usage directly lowers Ops/W.
• System Metric: Encompasses the efficiency of the processor, memory hierarchy, and data movement. Optimizing for bandwidth (via sparsity, fusion, caching) directly improves Ops/W.
• Commercial Importance: The primary metric for comparing edge AI accelerators (e.g., NPUs, GPUs) in battery-constrained devices.

Heterogeneous computing is a system architecture that integrates different types of processing units (e.g., CPUs, GPUs, NPUs, DSPs) to efficiently execute diverse workloads.

• Bandwidth Management: Different processors have different memory bandwidth capabilities and access patterns. Effective heterogeneous systems use a unified memory architecture or smart data placement to minimize costly data copies between processor memories.
• Edge AI Example: A system might use a low-power CPU for control logic, a high-bandwidth GPU for vision preprocessing, and an efficient NPU for model inference. Orchestrating data flow between them without saturating shared memory buses is critical.
• Optimization Goal: Assign computational tasks to the processor whose architecture (including bandwidth profile) best matches the task's requirements.