Graph Fusion

Each independent operator in a computational graph requires a separate kernel launch, which involves significant runtime overhead from the host CPU to the accelerator (NPU/GPU). This includes scheduling, argument passing, and synchronization. Graph fusion eliminates this overhead by merging multiple operators, resulting in a single launch for the fused block. This is especially impactful for models with many small, sequential operations, where launch overhead can dominate execution time.

A primary bottleneck in accelerator performance is moving data between global memory and compute cores. Without fusion, each operator writes its output tensor to global memory, only for the next operator to immediately read it back. Fusion creates a single kernel where intermediate results are passed directly between operators via fast registers or shared memory, bypassing slow global memory accesses. This dramatically reduces the memory bandwidth requirement and associated latency and power consumption.

By keeping intermediate tensor data within the high-speed memory hierarchy of the accelerator, fused kernels achieve superior data locality. This allows for more effective use of L1/L2 caches and reduces cache thrashing. Operations like element-wise additions followed by activations (e.g., Add → ReLU) can compute on data while it is still 'hot' in registers, avoiding the need to evict and reload data from slower memory levels.

Fusion allows compiler engineers to hand-optimize complex, compound operations that are perfectly tailored to the underlying hardware. For example, a fused Conv2D + BiasAdd + ReLU kernel can be written to use vendor-specific intrinsics and memory access patterns that would be impossible if the operators were separate. This unlocks peak FLOP/s and TOPS (Tera Operations Per Second) by maximizing hardware utilization and minimizing pipeline stalls.

High-level frameworks like PyTorch or TensorFlow have dispatching logic for each operator, checking device placement, data types, and selecting appropriate kernel implementations. Fusing operators reduces the number of dispatches the framework runtime must handle, lowering CPU overhead and simplifying the execution graph. This leads to more predictable latency and is a key enabler for ahead-of-time (AOT) compilation and deployment on resource-constrained edge devices.

A fused subgraph is treated as a single unit, enabling further compiler passes that are not possible on isolated operators. This includes:

• Constant folding within the fused block.
• Common subexpression elimination across operator boundaries.
• Aggressive layout transformations for the entire fused operation.
• Automatic kernel auto-tuning on the compound kernel as a whole. Fusion thus acts as a foundational optimization that creates larger, more optimization-friendly blocks for downstream passes.

Vendor SDKs perform final-stage graph fusion tailored to their silicon.

• NVIDIA TensorRT applies layer and tensor fusion to minimize kernel launches on its GPUs, combining operations like scale, clip, and activation.
• Apple Core ML tools fuse operations in the mlmodel graph specifically for Apple Neural Engine execution, optimizing for the unique memory hierarchy and execution units of ANE.

A precursor to fusion that groups sets of operators within a computational graph into logical clusters. These clusters are typically defined by hardware constraints (e.g., must fit on a specific core) or fusion heuristics. Clustering simplifies the fusion pass by identifying candidate subgraphs for merging.

• Purpose: To partition the graph for targeted optimization or distribution.
• Example: Grouping all convolution, batch norm, and ReLU operations into a single cluster destined for an NPU's tensor core.

A compiler optimization that identifies and eliminates redundant calculations of identical expressions within a computational graph. It replaces duplicate computations with a single stored value, reducing computational overhead. While graph fusion merges distinct operators, CSE removes identical copies of the same operator.

• Key Difference: Fusion combines different, adjacent ops; CSE removes identical, potentially non-adjacent ops.
• Benefit: Decreases FLOPs and kernel launch count, complementing fusion's memory bandwidth savings.

A classical compiler optimization that merges two or more adjacent loops iterating over the same range into a single loop. This improves data locality by keeping intermediate results in registers or cache, and reduces loop overhead. Graph fusion can be seen as a generalization of this concept from loop nests to dataflow graphs.

• Analogy: Loop fusion for for(i){A[i]=...} and for(i){B[i]=A[i]*...} is analogous to fusing element-wise operations in a graph.
• Hardware Impact: Both techniques aim to amortize memory access costs and improve instruction-level parallelism.

The strategy of dividing a large computational graph into smaller, manageable subgraphs. This is often done for distribution across multiple devices (e.g., model parallelism) or to fit subgraphs into limited on-chip memory. Graph fusion is typically applied within partitions to optimize each subgraph before partitioning decisions are finalized or after.

• Use Case: Splitting a large vision transformer model across four NPU chips.
• Interaction with Fusion: The partitioner must be aware of fusion opportunities, as fusing ops can change a subgraph's memory footprint and computational profile, affecting the optimal partition.

A low-level compiler pass that examines short sequences of instructions (a 'peephole') and replaces them with more efficient sequences. In graph compilation, this can manifest as pattern matching on small subgraphs (e.g., a Mul node followed by an Add node) and replacing them with a single, hardware-intrinsic Fused Multiply-Add (FMA) node. This is a form of fine-grained, local graph fusion.

• Scope: Local, instruction-level patterns.
• Example: Replacing (x * 2.0) + y with a single FMA(x, 2.0, y) operation in the graph IR.

A compiler pass that allocates memory buffers for all tensors in a computational graph, aiming to minimize peak memory usage. Graph fusion directly influences memory planning by eliminating the need for intermediate buffers between fused operators. Effective memory planning enables more aggressive fusion by ensuring fused kernels do not exceed available on-chip memory.

• Synergy: Fusion reduces buffer count; memory planning reuses buffers across non-fused ops.
• Critical for NPUs: Determines if large fused kernels can fit in fast SRAM or must spill to slower DRAM.