Fused Kernel

The fundamental goal of a fused kernel is to bypass the memory hierarchy bottleneck. In a standard, unfused execution pipeline, each primitive operator (e.g., a matrix multiplication followed by a bias add and ReLU) writes its full output tensor to global GPU memory (HBM) only for the next operator to immediately read it back. This store/load cycle consumes hundreds of clock cycles and saturates the memory bus.

A fused kernel keeps intermediate results in fast, on-chip memory:

• Registers: For scalar values and small vectors.
• Shared Memory: For tile-level intermediate tensors.
• L1/L2 Cache: For reusable data blocks. By fusing operations, the intermediate tensor never materializes in slow global memory, reducing DRAM bandwidth pressure and hiding memory latency.

Every GPU kernel launch incurs fixed launch latency and scheduling overhead from the host CPU and the GPU driver. For a model with hundreds of small, sequential operations, this overhead can constitute a significant portion of total runtime.

Fusion addresses this by:

• Reducing Launch Count: Converting N kernel launches into 1.
• Unified Resource Allocation: A single kernel manages its thread blocks, shared memory, and register file allocation once.
• Continuous Execution: Threads remain active, moving from one fused operation to the next without being disbanded and re-launched. This is particularly critical for lightweight, elementwise operations (like activations) where the launch cost can rival or exceed the computation time itself.

Arithmetic Intensity (AI) is a key hardware metric defined as the number of floating-point operations (FLOPs) performed per byte of data transferred from main memory (FLOPs/byte). Higher AI moves a workload from being memory-bound to compute-bound, better utilizing the GPU's massive parallel compute units.

Fusion directly increases AI by:

• Reusing Loaded Data: A single value loaded from global memory can be used by multiple fused operations (e.g., an input value used in a convolution, then batch norm, then activation).
• Fusing Light & Heavy Ops: Attaching low-FLOP elementwise ops (like GELU, SiLU) to heavy compute ops (like GEMM) "for free" on already-resident data. This transforms the kernel's profile, allowing it to approach the theoretical peak FLOPS of the hardware by keeping the compute units saturated with work.

Fused kernels are created through two primary methodologies:

Compiler-Driven Fusion (e.g., XLA, TVM, torch.compile):

• Graph-Level Analysis: The compiler's fusion planner analyzes the computational graph, identifying subgraphs (fusion groups) that are profitable to fuse.
• Pattern Matching: Recognizes common patterns like Conv -> BatchNorm -> ReLU.
• JIT/AOT Codegen: Generates the fused kernel code Just-In-Time at runtime or Ahead-Of-Time during compilation.

Hand-Written Fusion (Library Kernels):

• Manual Optimization: Experts write a single, highly-tuned CUDA/HIP/Metal kernel for a specific, performance-critical pattern.
• Canonical Examples: NVIDIA's cuDNN fused convolution kernels, or the FlashAttention kernel for multi-head attention.
• Extreme Optimization: Allows for deep, architecture-specific optimizations (e.g., warp-level operations, tensor core usage) that compilers may not yet automate.

Fusion strategies are categorized by the dataflow relationship between the operators being combined.

Vertical Fusion (Producer-Consumer):

• Fuses operators that are sequentially dependent in the graph.
• Example: LayerNorm -> Linear -> GELU. The output of one is the direct input to the next.
• Primary Benefit: Eliminates the intermediate tensor between the two stages.

Horizontal Fusion (Parallel/Sibling):

• Fuses operators that consume the same input or operate in parallel.
• Example: Two separate GELU operations applied to different branches of a model.
• Primary Benefit: Amortizes the cost of loading the shared input tensor across multiple operations, improving data reuse.

Modern compilers employ both strategies, often within the same kernel, to maximize data locality.

Fusion is not always beneficial. A compiler's cost model must evaluate fusion profitability.

Potential Downsides:

• Register Pressure: A fused kernel may require more live variables, exhausting the GPU's limited register file. This can force spilling to slower memory, reducing performance.
• Decreased Parallelism: Fusing a very large operation with a small one can cause thread divergence or underutilization of GPU cores.
• Compilation Complexity: The search space of possible fusions grows exponentially; the compiler must make near-optimal decisions quickly.

Profitability Heuristics consider:

• Operation Types: Elementwise ops are almost always profitable to fuse. Reductions are more complex.
• Data Size & Shape: Small, transient tensors are ideal candidates.
• Hardware Targets: Fusion strategies differ for data center GPUs (e.g., H100) versus mobile NPUs. The goal is to fuse as much as possible without triggering these negative side effects.

Operator fusion is the high-level graph optimization that identifies and merges adjacent computational nodes (operators) in a neural network's computational graph. It is the prerequisite step that defines what to fuse, which is then implemented by a fused kernel. The goal is to minimize intermediate tensor materialization in global memory.

• Graph-Level vs. Kernel-Level: Operator fusion works on the abstract graph; kernel fusion implements the fused graph node as efficient GPU code.
• Profitability Analysis: Compilers use cost models to decide if fusing two operators (e.g., a GeLU and Dropout) will improve performance by reducing memory bandwidth more than it harms occupancy or register usage.

Kernel launch overhead is the fixed latency and resource cost incurred each time the CPU instructs the GPU to execute a kernel. This includes API call handling, driver scheduling, and argument passing. Fused kernels directly target this bottleneck.

• Amortization: By combining many small operations into one kernel, the launch cost is paid once instead of many times.
• Significance on Modern Hardware: While absolute overhead is small (microseconds), in inference serving with continuous batching, where thousands of small, sequential operations are executed per request, the cumulative overhead becomes a major latency contributor.

Graph fusion is the automated process of applying operator fusion across an entire computational graph. It uses pattern matching and heuristics to identify fusion groups—subgraphs that are profitable to combine. This is a core pass in compilers like XLA, TVM, and MLIR.

• Pattern Matching: Compilers have built-in patterns for known beneficial fusions (e.g., Conv -> BatchNorm -> ReLU).
• Cost-Model Driven: Advanced compilers use a cost model for fusion to explore a search space of possible fusions, predicting the performance impact of each before finalizing the fusion plan.

These are two fundamental patterns for combining operators, distinguished by their data dependency structure.

• Vertical Fusion (Producer-Consumer): Merges sequentially dependent operators. Example: Fusing a matrix multiplication with a following bias addition. This eliminates the write of the intermediate matmul result to memory.
• Horizontal Fusion (Sibling Operations): Merges independent operators that consume the same input or operate in parallel. Example: Fusing the separate layer normalization operations applied to the query, key, and value projections in a transformer block. This improves compute density by reusing the input tensor.

Most real-world fusions, like Fused Multi-Head Attention, employ both patterns.

A fusion compiler is a specialized toolchain that performs end-to-end graph fusion and kernel generation. These are not monolithic applications but frameworks with specific fusion infrastructures.

• XLA (Accelerated Linear Algebra): Google's compiler, used by TensorFlow and JAX, known for aggressive ahead-of-time fusion (AOT Fusion). Its Fusion compiler pass is a primary optimization.
• TVM (Apache TVM): Uses its Tensor Expression language and Auto-Scheduler to generate highly optimized fused kernels for diverse hardware backends.
• MLIR (Multi-Level IR): Provides reusable dialects (like Linalg) and transformation passes that compiler engineers use to build their own fusion pipelines, offering more control than higher-level frameworks.

The primary performance goal of fusion depends on the limiting resource of the target operations.

• Memory-Bound Fusion: Applied when operations are limited by memory bandwidth (e.g., element-wise ops like Add, ReLU). The goal is fusion for cache—keeping data in fast registers or shared memory between ops to reduce trips to high-latency global memory (HBM).
• Compute-Bound Fusion: Applied to heavy arithmetic operations (e.g., large matrix multiplies). The goal is to increase arithmetic intensity (FLOPs per byte loaded). Fusing a light, memory-bound op (like bias add) onto a heavy compute-bound op can hide the light op's latency for free, as the data is already in cache.

Understanding this distinction is key to fusion profitability analysis.