Direct Memory Access (DMA)

The core function of DMA is to offload the task of moving data between memory and I/O devices (or between memory regions) from the Central Processing Unit (CPU). Instead of the CPU executing load/store instructions for each byte, a DMA controller manages the transfer. This frees the CPU to execute application logic or other tasks, dramatically improving overall system efficiency and throughput.

• Example: An NPU can initiate a DMA transfer of its input tensor from host RAM while the CPU continues to prepare the next batch of data.

DMA controllers are optimized to perform burst transfers, moving large, contiguous blocks of data in a single operation. This is far more efficient than single-cycle transfers because it amortizes the setup overhead of the memory bus transaction over a large amount of data. Burst mode maximizes memory bandwidth utilization and is essential for feeding data-hungry accelerators like NPUs and GPUs.

• Mechanism: The DMA controller acquires the system bus, sends the starting address, and then streams data for the entire block before releasing the bus.

Advanced DMA engines support scatter-gather operations. This allows a single DMA transaction to transfer data between a contiguous buffer in device memory and multiple non-contiguous (scattered) blocks in system memory, or vice versa. The DMA controller uses a descriptor list in memory that contains the source/destination addresses and sizes for each segment.

• Use Case: Gathering input features from disparate locations in host memory into a single contiguous tensor for an NPU kernel, or scattering an output tensor back to multiple non-contiguous result buffers.

Upon completion of a data transfer (or upon an error), the DMA controller generates a hardware interrupt to notify the CPU. This event-driven signaling is far more efficient than polling, where the CPU would repeatedly check a status register. The interrupt handler can then service the completed transfer (e.g., launch the next computational kernel on the NPU) with minimal latency.

• Flow: 1. CPU programs the DMA. 2. DMA executes transfer autonomously. 3. DMA raises an interrupt. 4. CPU interrupt handler processes completion.

DMA is not limited to device I/O. It supports several transfer types critical for accelerator workflows:

• Memory-to-Memory: High-speed copying or transformation of data between two regions of system RAM (e.g., aligning buffers).
• Memory-to-Device: The standard flow for loading model weights and input data from host memory into an NPU's local memory.
• Device-to-Memory: The flow for reading results from an NPU back to host memory.
• Device-to-Device: Direct transfer between two peripherals (e.g., between a network controller and an NPU), often bypassing host memory entirely.

For DMA to function efficiently with accelerators, the host system memory involved in the transfer must often be pinned (or page-locked). Pinning prevents the operating system's virtual memory manager from paging this memory to disk, guaranteeing it resides at a fixed physical address. This is a prerequisite because DMA controllers operate on physical addresses. Without pinning, the OS could move a page during a transfer, causing data corruption or system failure.

• Performance Trade-off: Pinning memory reduces the amount of memory available for OS paging, so it is typically used sparingly for high-performance data buffers.

Pinned memory (or page-locked memory) is host (CPU) system memory that the operating system is prevented from paging out to disk or moving. This is a mandatory prerequisite for high-performance DMA operations, as it guarantees the physical memory address of a data buffer remains stable, allowing a device's DMA controller to safely reference it for the duration of a transfer. Without pinning, the OS could relocate a page during a transfer, causing data corruption or system crashes.

• Key Use Case: Allocating host buffers for DMA transfers to/from GPUs, NPUs, network cards, and storage controllers.
• Performance Impact: Eliminates the need for the driver to create temporary pinned copies, reducing latency and CPU overhead for data preparation.

Memory-Mapped I/O (MMIO) is a method for a CPU to communicate with peripheral devices by mapping the device's control registers and data buffers into the CPU's physical address space. Instead of using special I/O instructions, the CPU uses standard load/store instructions to read from or write to these memory-mapped addresses, which are intercepted by the device. This is distinct from DMA, which is for bulk data transfer, but the two are closely related.

• Relationship to DMA: A CPU typically uses MMIO to write commands and parameters (like source/destination addresses) to a device's DMA controller to initiate a transfer. The DMA engine then performs the data movement autonomously.
• Example: Writing to a DMA_SRC_ADDR register via MMIO to tell an NPU where to fetch input tensor data from system memory.

Scratchpad memory (SPM) is a small, fast, software-managed on-chip memory (SRAM) explicitly controlled by the programmer or compiler, as opposed to a hardware-managed cache. In NPU architectures, SPM acts as a staging area for data being processed by the compute units. DMA is the primary mechanism for efficiently moving data between larger, slower external memory (like DRAM) and the high-performance scratchpad.

• DMA's Role: Programmers orchestrate double-buffering using DMA: while one batch of data in the SPM is being processed by the NPU cores, a DMA transfer is concurrently fetching the next batch from DRAM into a second buffer within the SPM, hiding memory latency.
• Advantage: Provides predictable, low-latency access and eliminates cache coherence overhead, making it ideal for streaming dataflows common in neural network layers.

Cache coherence is a property in multi-processor systems that ensures all caches have a consistent view of a given memory location. When a device like an NPU uses DMA to read from or write to memory that may also be cached by the CPU, coherence protocols must ensure data correctness. DMA is often non-coherent by default, meaning the device accesses memory directly, bypassing CPU caches.

• Coherent DMA: Some advanced interconnects (like CXL) support I/O coherency, allowing DMA engines to participate in the system's cache coherence protocol, simplifying software but adding hardware complexity.
• Non-Coherent DMA: The standard model. Software must explicitly flush CPU caches before a device reads data (to ensure it sees the latest writes from the CPU) and invalidate caches after a device writes data (to ensure the CPU doesn't read stale cached copies).

High Bandwidth Memory (HBM) is a 3D-stacked DRAM technology that provides extremely high data transfer rates in a compact, power-efficient form factor. It is commonly integrated on-package with high-performance accelerators like GPUs and NPUs as their primary working memory. DMA engines within the NPU are architected to saturate the massive bandwidth provided by HBM interfaces.

• DMA Design Impact: HBM's wide interfaces (1024-bit to 2048-bit) and high clock speeds demand sophisticated DMA controllers capable of sustaining transfer rates exceeding 1 TB/s. This often involves multiple, parallel DMA channels and deep buffers to maximize utilization.
• Use Case: Streaming weights and activations for large neural network models from HBM into the NPU's compute units via dedicated DMA paths, which is a primary bottleneck for model throughput.

Compute Express Link (CXL) is an open industry-standard interconnect that provides high-bandwidth, low-latency connectivity between the host CPU and devices like accelerators, memory expanders, and smart NICs. CXL builds upon the PCI Express physical layer but adds crucial memory semantics and optional cache coherence protocols. It fundamentally enhances DMA capabilities.

• Memory Semantics: CXL allows an NPU to access host memory as if it were its own local memory (load/store semantics), which can be more efficient than traditional DMA for small, random accesses. For large transfers, it still uses optimized DMA engines.
• Coherent DMA: CXL.io and CXL.mem protocols can enable I/O coherency, allowing the NPU's DMA to operate on memory that is cache-coherent with the CPU, dramatically simplifying software management of data shared between CPU and accelerator.