Operator Fusion

Each operation in a neural network graph typically requires launching a separate GPU or NPU kernel. This launch involves significant scheduling overhead from the host CPU to the accelerator. By fusing multiple operations, the framework launches a single, larger kernel, amortizing this fixed cost. This is critical for models with many small, sequential layers (e.g., activation functions following convolutions), where launch latency can dominate execution time.

Without fusion, the output tensor of one operation is written to global memory (e.g., GPU VRAM), only to be immediately read back as input for the next operation. This creates a memory bandwidth bottleneck. Fused kernels pass intermediate results directly through registers or shared memory within the same kernel, eliminating these costly round-trips to high-latency memory. This is especially beneficial for memory-bound operations and is a key technique for achieving peak arithmetic intensity.

Fusing operations keeps intermediate data "hot" in the processor's cache hierarchy. For example, fusing a convolution with a ReLU activation allows the convolution's output to be consumed by the ReLU while still in the L1/L2 cache, avoiding eviction. This temporal locality drastically reduces cache misses. Compilers use data dependency analysis to identify fusion candidates where producer-consumer relationships allow for such localized data flow.

Fusion is not merely concatenation; it allows for the creation of new, mathematically equivalent kernels that are more efficient than the sum of their parts. A classic example is fusing a matrix multiplication with a bias add and ReLU into a single GEMM+ReLU kernel. This fused kernel can:

• Use specialized instruction-level parallelism.
• Apply the ReLU non-linearity during the computation, avoiding a separate pass.
• Use kernel auto-tuning to find optimal tile sizes and thread mappings for the combined workload.

High-level frameworks like PyTorch or TensorFlow have per-operator dispatch logic to handle dynamic shapes, data types, and device selection. Executing a fused subgraph reduces the number of times this framework dispatch logic is invoked. In ahead-of-time (AOT) compilation scenarios, this overhead is eliminated entirely, as the fused kernel is compiled directly to native code. This leads to more predictable, low-latency execution critical for real-time inference.

Different hardware accelerators have unique characteristics that favor specific fusion patterns. A compiler's cost model evaluates these patterns for a target backend:

• GPUs: Fuse pointwise operations (ReLU, Sigmoid) with data-intensive ops to maximize thread occupancy and hide memory latency.
• NPUs: Fuse complex patterns like Conv-BatchNorm-ReLU into a single, hand-optimized micro-kernel that leverages dedicated systolic arrays.
• Mobile CPUs: Fuse operations to minimize power-intensive data movement between CPU cores and memory. This hardware-aware fusion is a key step in graph lowering for efficient deployment.

A compile-time optimization that evaluates and replaces expressions consisting entirely of compile-time constants with their precomputed result.

• Key Benefit: Eliminates runtime computation for static values, simplifying the graph.
• Example: Replacing tf.add(tf.constant(5), tf.constant(3)) with tf.constant(8).
• Relation to Fusion: Often performed before fusion to simplify patterns and reveal more fusion opportunities by removing intermediate constant nodes.

An optimization that identifies and eliminates redundant calculations of identical expressions within a computational graph.

• Mechanism: The compiler caches the result of a computed expression and reuses it wherever the same expression appears.
• Impact: Reduces FLOPs and memory traffic for repeated calculations.
• Contrast with Fusion: CSE reuses outputs; fusion merges distinct operations into a single kernel. They are complementary passes—CSE reduces work, while fusion reduces dispatch and memory overhead.

A compiler pass that identifies and removes operations whose outputs do not affect the final result of the program.

• Purpose: Prunes unused branches, unused outputs of ops, and entire subgraphs that are not connected to the final output.
• Performance Gain: Reduces both execution time and memory allocation for transient tensors.
• Prerequisite for Fusion: DCE is typically run early, cleaning the graph so fusion passes operate only on live, necessary operations.

A computation that overwrites its input tensor's memory buffer with its output, avoiding allocation of a separate buffer.

• Memory Benefit: Can significantly reduce peak memory footprint.
• Trade-off: Destroys the input data, which can break gradient computation if used carelessly during training.
• Synergy with Fusion: Fused kernels can be designed to perform in-place computation internally, combining the memory savings of in-place ops with the efficiency of a single kernel launch. Frameworks like PyTorch use in-place variants (e.g., relu_) for this purpose.

The process of transforming a high-level, hardware-agnostic intermediate representation (IR) into a lower-level, target-specific IR or machine code.

• Stages: A model may be lowered from a framework graph (e.g., PyTorch FX) to a compiler IR (e.g., MLIR), then to LLVM IR, and finally to GPU PTX or CPU assembly.
• Fusion Context: Operator fusion is a key transformation that often occurs during the lowering process, as the compiler decides how to map groups of high-level ops to efficient, hardware-specific kernels.

An automated process that searches for optimal implementation parameters for a computational kernel on specific hardware.

• Parameters Tuned: Tile sizes, loop unrolling factors, number of threads per block (CUDA), and vector width.
• Methodology: Uses empirical benchmarking, heuristic models, or machine learning to evaluate performance of many kernel variants.
• Critical for Fusion: The performance of a manually fused kernel is highly dependent on these parameters. Auto-tuning is essential for generating high-performance fused kernels across diverse hardware (e.g., different GPU architectures).