Kernel Fusion

Each kernel launch on a GPU or NPU incurs significant fixed overhead for scheduling, synchronization, and argument passing. By fusing multiple operations (e.g., a convolution, activation, and bias add) into one kernel, this overhead is paid once instead of multiple times. This is crucial for edge devices where latency is paramount and kernels may be small.

• Key Impact: Drastically reduces the scheduling latency that dominates short-running kernels.
• Example: Fusing three small kernels can cut total launch overhead by up to 66%.

A primary bottleneck in edge AI is memory bandwidth. Without fusion, each kernel writes its output to global memory (DRAM), which the next kernel must then read. This intermediate tensor movement consumes power and time.

Kernel fusion keeps data in fast on-chip memory (registers, shared memory, or cache) between operations.

• Key Impact: Eliminates costly round-trips to DRAM for intermediate results.
• Example: Fusing a GeLU activation onto a linear layer avoids writing the layer's output to memory and immediately reading it back.

When operations are separate, the data they produce may be evicted from the processor's cache before the next kernel can use it, forcing a reload from slower memory. A fused kernel operates on data while it is hot in the cache or even in registers.

This improves data reuse and reduces cache miss rates, which is critical for the constrained cache hierarchies of edge SoCs.

• Key Impact: Maximizes the utility of limited on-chip memory resources.
• Architectural Fit: Essential for systems with a memory-bound performance profile.

A compiler can perform more aggressive low-level optimizations on a single, larger fused kernel than on multiple small ones. This includes:

• Better instruction scheduling to hide memory latency.
• More effective loop unrolling and vectorization across the combined operation sequence.
• Reduced redundant address calculation for data accessed by multiple fused stages.

These optimizations increase Instructions Per Cycle (IPC) and better utilize the processor's execution units.

The combined benefits of reduced memory access and fewer kernel launches directly translate to lower energy consumption, a primary constraint for battery-powered edge devices.

• Memory Access = Power: DRAM accesses are among the most power-intensive operations on a chip. Minimizing them is a direct path to efficiency.
• Static Power Reduction: Consolidating work into a single kernel reduces the time the processor spends in idle states between kernels, lowering static power dissipation.

This makes kernel fusion a key technique for meeting Operations per Watt targets.

Kernel fusion is implemented within AI compilers and runtime frameworks. Key approaches include:

• Operator Fusion in Graph Compilers: Tools like Apache TVM, MLIR, and TensorRT analyze a model's computational graph and apply fusion patterns (e.g., vertical/horizontal fusion).
• Pattern Matching: Compilers identify common subgraph patterns (like Conv2D -> BiasAdd -> ReLU) and replace them with a single, hand-optimized or generated fused kernel.
• Just-In-Time (JIT) Fusion: Some runtimes can fuse kernels dynamically based on runtime input shapes and hardware capabilities.

Successful fusion requires deep knowledge of both the algorithmic semantics and the target hardware's execution model.

Edge-specific compilers apply constraint-aware fusion strategies that differ from data center optimizations. Key considerations include:

• Memory Pressure: Aggressive fusion can increase register usage and cause spilling; compilers must balance fusion depth with available on-chip memory.
• Power Budget: Fusing control-heavy and compute-heavy ops can prevent power-hungry components from idling.
• Real-Time Guarantees: For deterministic execution, fusion must not create kernels with highly variable execution times. Compilers may favor fission (splitting) for worst-case execution time (WCET) analysis.
• Heterogeneous Cores: Fusion decisions must account for which processor type (CPU, NPU, DSP) will execute the fused kernel.

A specific instance of kernel fusion applied at the graph level of a neural network. The compiler identifies sequences of primitive operations (e.g., Conv → BatchNorm → ReLU) that can be merged into a single, compound operator. This eliminates intermediate tensor materialization in memory.

• Example: Fusing a convolution, bias add, and activation function into one kernel.
• Impact: Reduces memory bandwidth pressure—a critical bottleneck on edge devices—and decreases kernel launch overhead.

The maximum rate at which data can be read from or written to a device's memory (e.g., DRAM) by the processor. It is measured in gigabytes per second (GB/s).

• The Bottleneck: Many edge AI workloads are memory-bound, not compute-bound. The time spent moving data often exceeds the time spent computing on it.
• Fusion's Role: Kernel fusion directly attacks this by minimizing the transfer of intermediate results between memory and the processor, thereby staying within the limited bandwidth envelope of edge hardware.

A classification of workloads based on what limits their performance.

• Compute-Bound: Performance is limited by the speed of the arithmetic logic units (ALUs). The processor is constantly busy with calculations.
• Memory-Bound (or Bandwidth-Bound): Performance is limited by the rate of data transfer to/from memory. The processor stalls waiting for data.

Kernel fusion primarily optimizes memory-bound sequences. By fusing ops, it increases operational intensity (operations per byte of data fetched), potentially moving the workload closer to being compute-bound and thus more efficient.

A property where many of the output values (activations) from a neural network layer are zero, often due to the use of ReLU or similar activation functions.

• Optimization Opportunity: Sparse activations create dynamic sparsity that can be exploited at runtime. A fused kernel can be designed to skip computations involving zero-valued activations entirely.
• Synergy with Fusion: Advanced compilers perform sparsity-aware kernel fusion, where the fused kernel code includes conditional logic to bypass zero-based computations, yielding further performance and power gains.

An analytical performance model used to visualize the attainable performance of a computational kernel. It plots performance (e.g., GFLOPs/sec) against operational intensity (FLOPs/byte).

• The Roofline: The plot shows two ceilings: a flat, memory-bound roof and a sloping, compute-bound roof.
• Tool for Analysis: Engineers use the roofline model to diagnose bottlenecks. Kernel fusion increases a kernel's operational intensity, moving its plotted point to the right. This can lift performance from the memory-bound plateau up towards the compute-bound ceiling, maximizing hardware utilization.