CUDA Graph

A primary benefit of CUDA Graphs is the elimination of repeated launch overhead. In a traditional CUDA workflow, each kernel launch or memory copy requires CPU intervention to manage the driver API, incurring latency. A CUDA Graph captures the entire workflow once. Subsequent executions launch the entire pre-defined graph with a single, low-latency CPU call (cudaGraphLaunch), bypassing the driver for each individual operation. This is critical for latency-sensitive applications like real-time inference.

The workflow involves two distinct phases:

• Capture: The application executes its normal CUDA sequence (using streams, events, and kernels) within a capture scope (cudaStreamBeginCapture / cudaStreamEndCapture). The CUDA driver records the kernels, dependencies, and parameters into a cudaGraph_t object.
• Instantiation: The captured graph is instantiated into an executable form (cudaGraphInstantiate). This process validates the graph and creates a cudaGraphExec_t object, which is the launchable entity. Instantiation has a one-time cost; the executable graph can be launched thousands of times with minimal overhead.

CUDA Graphs support efficient updates without re-capturing the entire graph. Kernel node parameters (e.g., pointers to input data) and memory copy node parameters can be updated in the executable graph (cudaGraphExecKernelNodeSetParams, cudaGraphExecMemcpyNodeSetParams). This allows the graph structure (the dependency graph) to remain static while dynamic data pointers or copy sizes can be modified. This is essential for batched inference where the model architecture is fixed, but input data changes per batch.

Advanced graph features enable dynamic control flow within the static graph structure:

• Conditional Nodes: Allow a subgraph to execute based on a runtime value computed on the device, enabling if-then-else logic within the graph.
• Event Record/Wait Nodes: Enable synchronization points within the graph, allowing parts of the graph to wait for an event recorded earlier in the same graph or in an external stream. This facilitates complex, inter-stream dependencies while maintaining the single-launch benefit.

During capture, the CUDA driver analyzes the true dependencies between operations. When instantiated and launched, the runtime can schedule independent kernels and memory copies with maximal overlap, often achieving better utilization than manually managed streams. The graph's internal scheduler has a complete, global view of all work, allowing for optimizations that are difficult to orchestrate manually. This reduces bubbles in the GPU's execution pipeline.

Many numerical algorithms (e.g., solvers, simulations, physics engines) consist of tight loops where the same kernel pattern executes thousands of times with slightly varying parameters. Examples include:

• Jacobi or Gauss-Seidel iterative solvers
• Conjugate Gradient methods
• Monte Carlo simulation steps Capturing this loop body as a graph eliminates the launch latency for every iteration. The graph is launched repeatedly from the GPU, with parameters updated via graph node updates or user-defined graph nodes, minimizing CPU involvement and maximizing GPU utilization.

Applications with strict real-time constraints, such as autonomous vehicle perception pipelines, robotic control loops, or high-frequency trading signal processing, cannot tolerate variable execution latency (jitter). The standard CUDA launch path involves unpredictable CPU scheduling and driver queue management. By using a pre-compiled CUDA Graph, the entire computational pipeline is submitted to the GPU as one atomic unit. This bypasses much of the driver's runtime scheduling, yielding sub-millisecond, deterministic launch times essential for hard real-time systems.

Advanced workloads like training or graphics rendering involve complex, staged pipelines with dependencies. Manually managing streams and events for overlap is error-prone. CUDA Graphs excel here by allowing the capture of an entire, dependency-resolved workflow spanning multiple CUDA streams. The graph's internal representation explicitly encodes kernel concurrency and memory copy dependencies. When launched, the GPU scheduler executes this pre-defined, optimal schedule every time, ensuring maximum hardware occupancy and efficient overlap of computation and data movement without runtime CPU analysis.

When profiling GPU kernel performance, the goal is to measure the true GPU execution time, isolated from CPU launch overhead and driver noise. Using CUDA Graphs provides a "clean" measurement by:

• Eliminating launch latency from the timed region.
• Ensuring identical execution traces across multiple runs for consistent results.
• Allowing precise measurement of kernel fusion benefits by capturing fused and unfused operation sequences in separate graphs for direct comparison. This is invaluable for performance architects optimizing low-level kernels.

While graphs are static, they support controlled dynamism for variable input sizes or conditional paths. Using graph node update APIs (e.g., cudaGraphExecKernelNodeSetParams), kernel parameters can be modified between launches. For conditional logic, conditional nodes or user-defined nodes can be embedded within the graph at capture time. This allows a single graph to handle a family of related workloads—like processing images of different resolutions or executing different model branches based on input—while still retaining most of the launch overhead benefits. The key is that the graph structure (the nodes and their dependencies) remains fixed.

Graph Compilation is the overarching process of transforming a high-level neural network computational graph into an optimized, hardware-specific sequence of low-level operations. This involves several stages:

• Operator Lowering: Decomposing framework-specific layers (e.g., a PyTorch Conv2d) into primitive tensor ops.
• Graph Optimization: Applying transformations like operator fusion, constant folding, and dead code elimination.
• Schedule Generation: Determining the precise order and parallel execution strategy for kernels. A CUDA Graph is the final, captured output of this process for NVIDIA GPUs—a static, executable representation of the optimized graph.

Operator Fusion is a critical compiler optimization that merges multiple sequential operations into a single, fused kernel. This is a primary optimization performed before CUDA Graph capture.

• Mechanism: Combines ops like Convolution, BatchNorm, and ReLU activation to create a single kernel.
• Benefit: Dramatically reduces kernel launch overhead and minimizes costly round-trips to global memory for intermediate results.
• CUDA Graph Role: The fused kernel sequence is captured as a single node in the graph. This eliminates the per-launch latency for each individual operation, compounding the performance gains of fusion.

Just-In-Time (JIT) Compilation is a strategy where code is compiled into machine instructions at runtime. This contrasts with the Ahead-Of-Time (AOT) approach used for CUDA Graphs.

• JIT (e.g., PyTorch Eager Mode): Each operation is dispatched individually, with drivers compiling kernels on-the-fly. This introduces overhead on every launch.
• AOT / CUDA Graph: The entire workflow is compiled and captured once. The resulting graph is launched repeatedly with near-zero CPU overhead.
• Trade-off: JIT offers dynamic flexibility, while CUDA Graphs provide maximal performance for static, repeatable workloads.

Kernel Launch Overhead refers to the latency incurred by the CPU driver stack each time it schedules a GPU kernel for execution. This overhead is the primary problem CUDA Graphs solve.

• Components: Includes argument marshaling, kernel validation, security checks, and scheduling onto the GPU's command queue.
• Impact: For workloads comprising many small kernels (common in machine learning), this overhead can dominate total execution time.
• CUDA Graph Solution: By capturing the entire sequence into a graph, the launch overhead is paid once during capture. Subsequent graph launches replay the pre-validated, scheduled sequence with minimal CPU involvement.

A CUDA Graph is fundamentally a recorded command buffer that can be replayed. This is its core execution mechanism.

• Capture Phase: All GPU operations (kernels, memory copies) are recorded into a graph object, creating a dependency-ordered command buffer.
• Graph Instantiation: The captured graph is instantiated into an executable graph, which is a hardened, ready-to-run form.
• Replay Phase: Launching the graph submits the entire pre-recorded command buffer to the GPU in a single, low-latency operation. The GPU driver replays the commands directly, bypassing the standard per-kernel launch pipeline.

CUDA Graphs represent a static graph execution paradigm, which differs from frameworks that use dynamic graphs.

• Dynamic Graph (Eager Mode): The execution graph is built on-the-fly by interpreting Python code. Operations are dispatched immediately, allowing for flexible control flow (e.g., if statements).
• Static Graph (CUDA Graph): The entire execution path must be known and fixed at capture time. Control flow must be unrolled, and tensor shapes must be constant.
• Optimization Implication: The static nature is what enables aggressive whole-program optimization, kernel fusion, and the elimination of launch overhead, but it requires the workload to be predictable and repeatable.