Profiling (CPU/GPU)

CPU Time measures the total time the CPU spends actively executing instructions for a specific function or operation, excluding time spent waiting for I/O or other processes. Wall Clock Time (or elapsed time) measures the total real-world time from start to finish, including all waits and overheads. The discrepancy between the two highlights I/O bottlenecks or contention for system resources.

• Example: A data preprocessing function may show low CPU time but high wall clock time, indicating it's stalled waiting for disk reads or network calls.

Profilers track the launch and duration of GPU kernels—the parallel functions executed on the GPU. Metrics include:

• Kernel Runtime: Time spent executing each CUDA or ROCm kernel.
• Occupancy: The ratio of active warps (thread groups) to the maximum supported by the streaming multiprocessor (SM), indicating utilization efficiency.
• Launch Overhead: Latency between kernel invocation on the CPU and execution start on the GPU. Inefficient, small kernels can be dominated by this overhead.

Tools like NVIDIA Nsight Systems or PyTorch Profiler visualize these timelines, showing which model layers trigger specific kernels.

A primary source of latency in GPU-accelerated workloads. Profilers measure:

• Host-to-Device (H2D) & Device-to-Host (D2H) Transfers: Time spent moving data between CPU (host) and GPU (device) memory over the PCIe bus. Excessive transfers are a common bottleneck.
• GPU Memory Allocation/Deallocation: Overhead from dynamic memory management APIs like cudaMalloc.
• GPU Kernel Memory Accesses: Profilers can identify memory-bound kernels where execution time is limited by the speed of reading/writing to GPU global or shared memory, versus compute-bound kernels limited by arithmetic logic unit (ALU) throughput.

Deep learning profilers decompose the model's execution graph and time individual operators (ops). This reveals which layers consume the most inference time.

• Attention vs. FFN Layers: In transformers, profile the time for multi-head attention versus feed-forward network blocks.
• Convolution Parameters: For CNNs, profile by kernel size, stride, and input/output channels.
• Framework Overhead: Time spent in the framework's Python-to-C++ dispatch layer versus the core compute kernel.

PyTorch Profiler with TensorBoard provides flame graphs showing time spent per operator, such as aten::matmul or aten::convolution.

Profilers intercept calls to the CUDA driver and runtime APIs to quantify their latency contribution.

Key traced calls include:

• cudaMemcpyAsync: For asynchronous memory transfers.
• cudaLaunchKernel: For launching GPU kernels.
• cudaStreamSynchronize: Time spent waiting for a stream of operations to complete.
• cudaMalloc / cudaFree: Dynamic memory management overhead.

Frequent, small API calls can create significant CPU-side overhead, limiting overall throughput. Profiling helps batch operations or adjust stream usage to minimize this.

Advanced profilers correlate application metrics with system-wide resource utilization to identify external bottlenecks.

• CPU Utilization: Per-core usage to detect single-threaded bottlenecks or excessive context switching.
• GPU Utilization: The percentage of time the GPU's engines (graphics, compute, copy) are active.
• PCIe Bandwidth: Saturation of the bus connecting CPU and GPU.
• Power & Thermal Throttling: Events where the GPU or CPU reduces clock speed due to temperature or power limits, causing sudden latency spikes.

Tools like NVIDIA DCGM (Data Center GPU Manager) or Linux perf provide this system-level view alongside application traces.

The total time delay between submitting an input to a machine learning model and receiving its output. Profiling decomposes this latency into constituent parts:

• Prefilling: Processing the static prompt.
• Decoding: Autoregressive token generation.
• GPU Kernel Execution: Time spent in compute kernels.
• Memory Transfers: Data movement between CPU and GPU (PCIe overhead).

The primary goal of profiling. It involves analyzing execution traces to pinpoint the limiting resource or operation. Common bottlenecks include:

• GPU Compute-Bound: Saturation of streaming multiprocessors (SMs).
• Memory-Bound: Bandwidth limitations on GPU VRAM or system RAM.
• Kernel Launch Overhead: Excessive latency from frequent, small kernel dispatches.
• CPU Serialization: Host-side preprocessing or data loading blocking the GPU.

The fixed latency cost of scheduling a computational kernel on the GPU. Profilers like NVIDIA Nsight Systems measure this directly. This overhead is critical for small operations and can be mitigated by:

• Operator Fusion: Combining sequential ops into a single kernel.
• Kernel Consolidation: Rewriting model code to use fewer, larger kernels.
• Using CUDA Graphs: Capturing a sequence of kernels into a single, replayable graph to eliminate repeated launch overhead.

A compiler-level optimization identified and validated through profiling. It merges multiple sequential neural network operations into a single GPU kernel. Benefits include:

• Reduced Kernel Launches: Lowers GPU kernel launch overhead.
• Fewer Memory Accesses: Intermediate tensors are kept in registers/cache instead of written to global memory.
• Framework Support: Automatically performed by TensorRT, ONNX Runtime, and PyTorch's torch.compile with Inductor backend.

An optimized, static representation of a neural network's computational operations produced by inference compilers. Profiling often targets the execution of this graph. Key aspects:

• Static Optimization: Enables ahead-of-time operator fusion, kernel selection, and memory planning.
• Profiler Integration: Tools like PyTorch Profiler can trace execution at the granularity of graph nodes.
• Runtime Overhead Reduction: Eliminates interpreter overhead present in eager execution modes.