GPU Direct Storage (GDS)

GDS's foundational feature is the Direct Memory Access (DMA) engine bypass. In a traditional I/O path, data must travel from storage to the CPU's system memory via the PCIe bus, and then be copied again to GPU memory via a DMA engine. GDS establishes a direct memory access (DMA) path from the NVMe storage device to the GPU's memory, eliminating the intermediate bounce through host RAM. This reduces latency by removing a full memory copy and lowers CPU utilization by offloading the data movement task from the CPU to the storage and GPU hardware.

• Traditional Path: NVMe SSD -> PCIe -> Host RAM -> DMA Engine -> PCIe -> GPU RAM.
• GDS Path: NVMe SSD -> PCIe -> GPU RAM.

This feature leverages PCIe peer-to-peer (P2P) capabilities. Normally, all PCIe traffic is routed through the root complex (typically the CPU). GDS enables a direct peer-to-peer transaction between the NVMe controller (a PCIe device) and the GPU (another PCIe device) on the same PCIe switch fabric. The data transfer is managed by the two endpoint devices without involving the root complex. This requires system BIOS and platform support but is critical for achieving the lowest possible latency and freeing the CPU from I/O management duties for data-intensive workloads like training on massive datasets or loading large neural network checkpoints.

The cuFile API is the software interface for GDS. It provides a set of CUDA library calls (e.g., cuFileRead, cuFileWrite) that applications use to initiate direct transfers. A key feature is its compatibility mode, which allows applications to use the same API calls regardless of whether the underlying system has GDS hardware support. If GDS is unavailable, the calls fall back to a standard, CPU-buffered path seamlessly. This ensures developer adoption isn't gated by hardware requirements. The API integrates with standard file I/O libraries like fopen/fread via driver-level interception, allowing legacy applications to benefit from GDS with minimal code changes when linked against the cuFile library.

While GDS aims to bypass host memory, certain operations or system configurations may still require its use. GDS Pinned Memory (or GDS-buffer ) is a specialized allocation of page-locked host memory that is optimized for use with the GDS driver. When data must temporarily reside in host memory, using this specially allocated buffer minimizes driver overhead and ensures optimal alignment for the remaining DMA transfers to the GPU. It is more efficient than standard cudaMallocHost for GDS workflows. This is often used during the compatibility fallback path or when preprocessing data on the CPU before a final transfer to the GPU.

GDS operations are designed to be asynchronous and non-blocking. An application can issue a cuFileReadAsync call, which returns control to the program immediately while the direct data transfer occurs in the background on the PCIe bus. This allows the GPU's compute engines (SMs) to continue processing other data concurrently with the I/O operation, effectively overlapping I/O and compute. This is managed using CUDA streams, where I/O tasks and compute kernels are enqueued into the same or different streams, enabling fine-grained control over concurrency and dependencies. This overlap is essential for hiding the latency of loading the next batch of data in a training pipeline.

GDS is designed for scalable, multi-GPU systems. A single storage namespace (e.g., a large NVMe array) can serve data directly to multiple GPUs simultaneously. Each GPU can establish its own independent DMA path from storage. When combined with a high-bandwidth storage fabric (like NVMe-oF), this allows aggregate I/O bandwidth to scale nearly linearly with the number of GPUs, preventing the storage subsystem from becoming a bottleneck in large-scale training clusters. This feature is critical for distributed data-parallel training, where each GPU needs rapid access to different shards of the global dataset.

Scientific computing and analytics workloads involving massive arrays benefit from GDS. It allows GPUs to process data in-place on storage or load only necessary chunks, crucial for:

• Computational Fluid Dynamics (CFD) and finite element analysis with huge mesh files.
• Genomic sequence analysis where reference genomes and read data are voluminous.
• Financial risk modeling using multi-dimensional time-series data.
• Climate and weather simulation data post-processing and visualization.

For inference tasks requiring context beyond GPU memory—like long document analysis, video processing, or enterprise RAG—GDS enables dynamic, on-demand data fetching. This supports:

• Retrieval-Augmented Generation (RAG) with large, proprietary knowledge bases stored on fast SSDs.
• Long-context language models that need to access relevant document segments without full pre-loading.
• Multi-modal inference combining large image, video, and text contexts.

Processing high-resolution, high-frame-rate video streams or large image datasets requires sustained high bandwidth. GDS provides a direct data path from storage to GPU compute engines for:

• AI-powered video analytics (e.g., object detection, facial recognition) on archived footage.
• Content creation and rendering where assets like textures and 3D models are streamed.
• Medical imaging pipelines analyzing high-resolution MRI or CT scans directly from archival storage.

GPU-accelerated databases and data processing frameworks use GDS to bypass host memory, reducing query latency. This is key for:

• Apache Spark RAPIDS and other GPU-accelerated SQL engines.
• Parquet/ORC file processing where columnar data is filtered and aggregated on the GPU.
• Vector database operations where nearest neighbor searches are performed on embeddings stored on NVMe.

GDS is not used in isolation. Its effectiveness depends on and complements other GPU memory optimization techniques:

• Unified Virtual Memory (UVM): GDS can populate GPU memory pages on-demand, integrated with UVM's page faulting mechanism.
• Peer-to-Peer (P2P) over NVLink: Data loaded via GDS can be directly shared between GPUs.
• Memory Tiering: GDS acts as the fastest tier for storage, integrated into a hierarchy with GPU HBM and CPU RAM.
• Demand Paging: GDS operations are often triggered by GPU page faults for non-resident data.

A memory management architecture that creates a single, contiguous virtual address space shared between a CPU and GPU. This allows both processors to access data using the same memory pointers, simplifying programming by eliminating the need for explicit data copies. UVM enables on-demand page migration, where memory pages move between CPU and GPU memory based on access patterns.

• Key Benefit: Simplifies data sharing and enables memory oversubscription.
• Relation to GDS: While UVM manages a unified view of CPU and GPU memory, GDS extends this paradigm by allowing the GPU to directly address storage, creating a more expansive virtual address space that includes NVMe devices.

A technique where a GPU accesses data directly from host (CPU) memory without performing an explicit copy into GPU device memory. This reduces transfer latency and host memory pressure. It is typically enabled by using page-locked (pinned) host memory and relies on the PCIe bus for access.

• Key Benefit: Eliminates a memory copy, reducing latency for data already in host memory.
• Contrast with GDS: Zero-copy operates between host RAM and GPU. GDS bypasses host RAM entirely, allowing the GPU to pull data directly from an NVMe SSD. GDS is for storage-to-GPU transfers, while zero-copy is for host-to-GPU transfers.

A capability that allows multiple GPUs within the same system to directly read from and write to each other's device memory over a high-speed interconnect like NVLink or PCIe, bypassing the need to stage data through host memory.

• Key Benefit: Dramatically increases bandwidth and reduces latency for multi-GPU communication.
• Relation to GDS: Both technologies are forms of direct data access that remove the CPU and host memory as bottlenecks. P2P enables GPU-to-GPU direct access, while GDS enables storage-to-GPU direct access. They can be combined in workflows where data moves from storage directly to one GPU, then via P2P to others.

NVIDIA's high-bandwidth, energy-efficient interconnect technology. It enables direct communication at speeds significantly higher than traditional PCIe.

• CPU-GPU NVLink: Creates a coherent memory space between CPU and GPU, similar to UVM but with much higher bandwidth.
• GPU-GPU NVLink: Forms the physical backbone for high-performance Peer-to-Peer (P2P) access.
• Role in GDS Ecosystem: While GDS uses the PCIe bus for storage access, NVLink is critical for the subsequent high-speed movement of that data between GPUs or between the GPU and a NVLink-connected CPU, ensuring the full pipeline remains optimized.

Core mechanisms of Unified Virtual Memory (UVM) that enable memory oversubscription.

• Demand Paging: Data is transferred from a slower backing store (like system memory or an SSD) into faster GPU memory only when the GPU attempts to access it.
• GPU Page Fault: The exception raised when this access occurs, triggering the memory management unit to migrate the required page.
• Integration with GDS: GDS can act as the backing store for this process. When a GPU page fault occurs for data not in CPU or GPU memory, the GDS driver can service the fault by directly loading the required data blocks from the NVMe SSD into GPU memory, streamlining the path.

The physical memory technology on modern high-performance GPUs. HBM stacks DRAM dies vertically next to the GPU die using through-silicon vias (TSVs), offering extreme bandwidth and improved energy efficiency compared to traditional GDDR memory.

• Key Characteristic: Provides the ultra-fast memory where compute happens.
• System Bottleneck: The immense compute power fed by HBM can be starved by slow I/O from storage. GDS directly addresses this bottleneck by providing a high-bandwidth pipeline (via PCIe) from storage to HBM, ensuring the GPU's fast memory is kept saturated with data for processing.