CUDA

Introduced to simplify programming, Unified Memory creates a single virtual address space accessible by both the CPU and GPU. The driver and hardware automatically migrate data pages on demand, triggered by page faults. This eliminates explicit cudaMemcpy calls for complex data structures. On modern systems with Page Migration Engine support, data can be physically resident on either the CPU or GPU, and the system transparently moves it to the accessing processor, enabling oversubscription of GPU memory with system RAM.

CUDA Graphs capture a sequence of kernel launches and memory operations as a single, replayable, directed acyclic graph. This drastically reduces the CPU launch overhead for repetitive workloads by submitting the entire graph to the GPU in a single operation. Key benefits include:

• Reduced latency: Eliminates per-kernel launch overhead.
• Static optimization: The CUDA driver can optimize the graph's execution plan offline.
• Deterministic execution: Ensures consistent timing for safety-critical or real-time applications. This is essential for low-latency inference where the same model graph is executed repeatedly.

The NVIDIA Collective Communications Library provides topology-aware multi-GPU and multi-node communication primitives optimized for CUDA. It implements routines like all-reduce, all-gather, and broadcast with ring and tree algorithms. NCCL automatically detects the fastest interconnect—NVLink, InfiniBand, or RoCE—and uses GPUDirect RDMA to bypass the CPU, achieving near-hardware-limit bandwidth for distributed training frameworks like PyTorch Distributed.

An SDK that optimizes trained neural networks for deployment on NVIDIA GPUs. TensorRT performs layer fusion, kernel auto-tuning, and precision calibration (FP32, FP16, INT8) to maximize throughput and minimize latency. It parses models from frameworks like TensorFlow and PyTorch, then generates a highly optimized runtime engine. Key features include dynamic tensor memory and concurrent stream execution for production serving.

A hardware partitioning feature on NVIDIA A100, H100, and H200 GPUs that divides a single physical GPU into up to seven isolated instances. Each instance has dedicated compute cores, memory, and cache, with guaranteed quality of service. MIG enables secure multi-tenant GPU sharing without virtualization overhead, allowing concurrent CUDA workloads from different users or containers to run with deterministic performance and fault isolation.

A family of technologies enabling direct data paths between GPUs and third-party devices. GPUDirect RDMA allows network adapters to read/write GPU memory directly, bypassing CPU and system memory for ultra-low-latency inter-node communication. GPUDirect Storage creates a direct path between NVMe drives and GPU memory, enabling asynchronous I/O that keeps the GPU compute pipeline saturated during data-intensive training.

A CUDA stream is a sequence of operations that execute in issue-order on the GPU. By creating multiple streams, developers can overlap kernel execution with data transfers (memcpy) and achieve concurrent execution of independent kernels. This is critical for maximizing GPU utilization—while one stream computes, another transfers the next batch. Streams are managed via cudaStream_t and synchronized with events.

The Data Center GPU Manager is a suite of tools for monitoring, diagnosing, and managing NVIDIA GPUs in cluster environments. It exposes telemetry on GPU utilization, temperature, power draw, ECC errors, and PCIe throughput. DCGM integrates with Kubernetes via the GPU Operator and feeds metrics into Prometheus/Grafana stacks. Its diagnostic engine runs active tests to detect hardware anomalies before they cause silent data corruption.