Kernel Launch Overhead

Every kernel launch requires a series of synchronous calls from the application (e.g., via CUDA or ROCm) down to the hardware driver. This involves:

• Parameter marshaling: Packing kernel arguments, grid/block dimensions, and shared memory configuration into a launch configuration.
• State validation: The runtime validates the launch parameters, checks resource availability (registers, shared memory), and ensures memory consistency.
• Driver submission: The validated launch packet is enqueued to the GPU's command queue via the driver's system call interface (ioctl on Linux). This sequence, though optimized, introduces microsecond-scale latency per launch that is pure overhead.

The GPU is a shared resource. Launching a kernel often requires implicit synchronization points:

• Stream synchronization: Kernels within the same CUDA stream execute sequentially. The runtime must ensure prior kernels on the stream have completed, which may involve waiting for a GPU fence.
• Context switching: If the GPU is managing multiple processes (e.g., in a multi-tenant inference server), a kernel launch may require a context switch—saving and restoring the GPU's execution state for the current process. This is a heavy operation that can take hundreds of microseconds.
• Default stream overhead: Kernels launched on the legacy default stream implicitly synchronize with all other streams, creating a global barrier and serializing execution.

When a new kernel is launched, the GPU's instruction processing pipeline is cold:

• Kernel code must be loaded from device memory into the instruction cache (I-cache). This fetch can stall the GPU's streaming multiprocessors (SMs).
• Branch prediction buffers are empty, leading to initial mispredictions for conditional logic within the kernel.
• Warp schedulers need to discover the execution pattern of the new kernel's warps to achieve optimal dual-issue scheduling. This warm-up phase represents lost compute cycles. Fusing kernels amortizes this cost over more computational work.

Each discrete kernel launch acts as a memory barrier in the GPU's hierarchy, limiting data locality:

• L1/Shared Memory: Data in fast on-chip caches is typically not preserved between kernel launches. A subsequent kernel must reload data from slower global memory (HBM).
• Constant Cache: The constant cache, used for kernel parameters and broadcast values, is often invalidated.
• Register File: The register state is completely cleared between kernels. Intermediate results that could be held in registers must be written to and read from global memory. Fusion keeps intermediate tensors in registers or shared memory, turning memory-bound operations into compute-bound ones.

Configuring the execution grid for a kernel has non-trivial cost:

• Grid/Block Dimension Calculation: The runtime must compute and validate the mapping of thread blocks to streaming multiprocessors.
• Resource Allocation: The launch system must allocate registers and shared memory per thread block statically before execution. Over-provisioning (e.g., requesting too much shared memory) can reduce occupancy, while under-provisioning is a hard launch failure.
• Dynamic Parallelism Overhead: If a kernel launches child kernels (dynamic parallelism), the setup and management of the nested grid introduces significant additional latency and complexity, often orders of magnitude greater than a top-level launch.

Launch latency is the pure time delay between when a host CPU instructs the GPU to execute a kernel and when the first thread of that kernel begins execution on the GPU.

• Components: Includes driver overhead, command buffer submission, and GPU scheduler dispatch.
• Typical Magnitude: On the order of 5-50 microseconds per kernel, which is significant for kernels that themselves execute in microseconds.
• Measurement: A key component of the broader kernel launch overhead, which also includes resource setup costs.

Fusion profitability is the analysis of whether combining a set of operators will result in a net performance gain, weighing reduced overhead against potential downsides.

• Benefits: Reduced launch overhead, improved data locality, lower global memory bandwidth usage.
• Costs: Increased register pressure per thread, reduced occupancy, more complex kernel code.
• Compiler Decision: Driven by a cost model for fusion that estimates execution time of fused vs. unfused paths based on hardware characteristics and operator properties.