Operation Fusion

The primary benefit of operation fusion is the elimination of intermediate memory traffic. Without fusion, the output of one kernel (e.g., a convolution) must be written to global memory (DRAM) before being read back by the next kernel (e.g., a bias add). This creates a memory-bound bottleneck. Fusion keeps intermediate tensor results in fast, on-chip memory hierarchies like registers or shared memory, drastically reducing accesses to slower off-chip memory. This is critical for operations with low arithmetic intensity, where performance is limited by memory bandwidth.

Each individual kernel launch on an NPU or GPU incurs non-trivial scheduling overhead. This includes:

• Driver and runtime API calls
• Argument marshaling and setup
• Workload distribution to streaming multiprocessors By fusing a sequence like Conv2D -> BiasAdd -> ReLU into one kernel, multiple launches are replaced by one. This reduces latency and CPU involvement, improving overall throughput, especially for networks with many small, sequential operations.

Fusion enhances temporal locality. When operations are separate, an intermediate tensor may be evicted from cache before the next kernel consumes it. The fused kernel operates on the data while it is hot in the cache or registers. This principle is similar to loop fusion in traditional compilers. It allows for more effective use of the NPU's memory hierarchy, minimizing costly cache misses and keeping the computational units fed with data.

A fused kernel presents a larger, unified code block to the downstream compiler, enabling aggressive low-level optimizations that are impossible across kernel boundaries. These include:

• Common subexpression elimination across the original operation boundaries
• More effective register allocation and reduced register spilling
• Improved instruction-level parallelism (ILP) through scheduling
• Kernel vectorization across the combined operation flow This creates a synergistic effect where fusion unlocks deeper optimization potential.

Reducing memory accesses has a direct, positive impact on power consumption. DRAM accesses are orders of magnitude more energy-intensive than arithmetic operations or on-chip memory accesses. By minimizing data movement—the dominant consumer of energy in modern AI chips—operation fusion significantly improves operations per watt. This is essential for edge AI and mobile deployment where thermal and power budgets are severely constrained.

Operation fusion is routinely applied to predictable, sequential patterns in neural networks. Key examples include:

• Linear/Bias/Activation: MatMul + BiasAdd + GELU/ReLU
• Batch Normalization Fusion: Folding batch norm parameters into preceding convolution weights during inference.
• Convolution Fusion: Conv2D + BiasAdd + Activation + (optional Residual Add).
• Element-Wise Operation Chains: Sequences like Add -> Multiply -> Sigmoid. Frameworks like TensorFlow/XLA and PyTorch/TorchInductor automatically identify and fuse these patterns during graph compilation.

The most common pattern, fusing consecutive elementwise operations (e.g., addition, multiplication, activation functions) that are applied independently to each element of a tensor. This pattern eliminates the need to write the intermediate tensor to global memory between each operation.

• Example: output = relu(bias_add(convolution(input, weights), bias))
• Key Benefit: Transforms multiple memory-bound kernels into a single compute-bound kernel, drastically reducing memory bandwidth pressure.

Fuses a reduction operation (e.g., sum, max) with a preceding producer kernel. The reduction is performed on intermediate values while they are still in fast registers or shared memory, avoiding a full round-trip to global memory.

• Example: Fusing a matrix multiplication's output with a column-wise sum for layer normalization.
• Key Benefit: Critical for memory-bound reductions, as the intermediate data size is often large. This pattern is central to optimizing operations like Softmax and LayerNorm.

Fuses an operation that requires broadcasting a smaller tensor (e.g., a bias vector, a scaling factor) with a preceding computation. The broadcasted values are combined with elementwise operations in-register.

• Example: Fusing a per-channel bias addition and scaling directly into a convolution's output loop.
• Key Benefit: Eliminates the materialization and repeated reading of the broadcasted tensor, saving memory bandwidth and capacity in on-chip caches.

Fuses operations from adjacent neural network layers into a single kernel. This is a higher-level, graph-based optimization that requires matching tensor dimensions and data dependencies across layer boundaries.

• Example: Fusing a Convolution, Batch Normalization, Bias Add, and ReLU activation from a single block in a ResNet.
• Key Benefit: Dramatically reduces latency for end-to-end inference by minimizing kernel launch overhead and inter-layer data movement, a key target for graph compilers like TVM, XLA, and MLIR.

Fuses independent operations that consume the same input data into a single kernel. Multiple output tensors are computed in parallel within the same loop structure.

• Example: A single kernel that computes both the mean and variance of a tensor in one pass for Batch Normalization statistics.
• Key Benefit: Improves arithmetic intensity and memory access efficiency by reusing the loaded input data for multiple computations, amortizing the memory access cost.

A canonical deep learning fusion that combines a matrix multiplication (MatMul), an elementwise bias addition, and the non-linear Gaussian Error Linear Unit (GELU) activation. This pattern is ubiquitous in transformer models (e.g., within feed-forward networks).

• Mechanism: The kernel computes the MatMul result for a tile, adds the broadcasted bias while the data is in registers or shared memory, and then applies the GELU approximation formula before writing the final result to global memory.
• Performance Impact: This fusion can provide a >2x speedup compared to executing three separate kernels, by avoiding two intermediate writes/reads of large tensors.

Kernel fusion is a compiler optimization that merges multiple, separate computational kernels—each representing a distinct function call on the accelerator—into a single, larger kernel. This eliminates the overhead of repeated kernel launches and the mandatory synchronization and data transfers between global memory that occur between kernels. It is a broader, more coarse-grained optimization than operation fusion, often applied at the level of entire functions or layers.

• Primary Benefit: Reduces launch latency and global memory traffic.
• Scope: Applied across function or layer boundaries.
• Example: Fusing a standalone normalization layer kernel with a subsequent activation layer kernel.

Loop fusion is a compiler transformation that combines two or more adjacent loops which iterate over the same range into a single loop. This improves data locality by keeping intermediate results in faster memory hierarchies like registers or caches, as fused data is consumed immediately in the same loop iteration. It also reduces loop control overhead (e.g., increment and branch instructions).

• Primary Benefit: Enhances cache locality and reduces loop overhead.
• Scope: Applied at the source code or intermediate representation (IR) level within a single kernel.
• Contrast with Operation Fusion: Loop fusion works on loop structures; operation fusion works on primitive tensor operations, which may already be inside fused loops.

Common subexpression elimination is a compiler optimization that identifies redundant computations of identical expressions within a program's scope. It computes the expression once, stores the result in a temporary variable, and replaces all subsequent occurrences with a reference to that variable. This reduces computational cost and can decrease register pressure.

• Primary Benefit: Eliminates redundant arithmetic operations.
• Scope: Local basic blocks or entire functions.
• Relation to Fusion: While fusion combines distinct operations, CSE removes duplicate instances of the same operation. They are complementary optimizations.

Dead code elimination is a compiler pass that removes code which does not affect the program's output. This includes:

• Unreachable code: Statements that can never be executed.
• Dead stores: Writing values to variables that are never read.
• Useless computations: Operations whose results are not used.

This optimization reduces binary size, execution time, and resource usage (e.g., registers). It is often enabled by other transformations like constant propagation and operation fusion, which can render previously live code obsolete.

Constant propagation is an optimization that replaces variables known to have constant values with those literal values. Constant folding is the subsequent evaluation of expressions involving only constants at compile time. For example, the expression X = 3 * 5 is folded to X = 15.

• Primary Benefit: Reduces runtime computation and enables further optimizations like dead code elimination.
• Impact on Fusion: Can simplify the computational graph by resolving static values, making patterns for operation fusion (e.g., fusing a constant bias addition) more apparent and trivial to implement for the compiler.

Peephole optimization is a low-level, pattern-matching compiler technique that examines short sequences of generated instructions (a "peephole") and replaces them with more efficient sequences. It operates on the final assembly or machine code IR.

• Common Replacements: Strength reduction (e.g., x * 2 → x << 1), redundant load-store elimination, and dead instruction removal.
• Scope: Very local, typically a few consecutive instructions.
• Relation to Fusion: While operation fusion is a high-level graph transformation, peephole optimizations clean up and perfect the low-level code that results from such transformations, ensuring the fused kernel's instructions are themselves optimal.