Operator Fusion

The primary benefit of fusion is eliminating intermediate memory writes and reads. In a sequence like Convolution → Bias Add → ReLU, the output of each operation is typically written to global memory before being read by the next. Fusion creates a single kernel where the intermediate results are passed directly through registers or shared memory, drastically reducing DRAM bandwidth pressure. This is quantified by the Roofline Model; fusion increases the kernel's operational intensity, moving it from being memory-bound toward being compute-bound.

Each independent GPU/NPU kernel launch incurs significant overhead from the host CPU driver. This includes:

• Scheduling and dispatch latency.
• Setting up kernel arguments and memory addresses.
• Potential synchronization points between kernels. By fusing multiple operations, the compiler reduces the number of discrete kernel launches. This minimizes CPU-GPU synchronization and allows the hardware's thread schedulers to maintain higher occupancy, leading to better utilization of parallel execution units.

Compilers like TVM, XLA, and vendor SDKs (e.g., TensorRT, OpenVINO) perform fusion by identifying specific subgraph patterns in the neural network's computational graph. Common fusible patterns include:

• Element-wise ops: ReLU, Sigmoid, Add.
• Broadcast ops: Bias addition, scaling.
• Reduction ops: Sum, Mean (when combined with preceding ops). The compiler's graph optimizer scans for these predefined or learned patterns and replaces the matched subgraph with a single, custom-fused kernel implementation from a kernel library or generates one via auto-tuning.

At the code generation level, fusion is implemented through loop fusion. Separate loops for each operation are merged into a single, larger loop nest. For example, the loops for computing a convolution and applying a ReLU are combined, allowing the ReLU to be applied immediately to each output element as it is computed, before it is ever written to a temporary array. This requires the compiler to analyze data dependencies and loop bounds to ensure correctness. The final fused kernel code is then optimized with techniques like loop tiling and vectorization for the target NPU.

Advanced NPUs often provide dedicated, fixed-function hardware units or complex instructions (intrinsics) for common fused operations. For instance, a NPU may have a single instruction for a Fused Multiply-Add-Activation. The compiler's operator lowering phase must map the identified fusible pattern directly to this hardware intrinsic. This mapping is the most efficient form of fusion, as it leverages silicon designed explicitly for the combined operation, offering optimal performance and power efficiency. This is a key part of vendor SDK and intrinsic mapping.

Not all operations can be fused. Key constraints include:

• Data dependencies: Operations must be sequentially dependent.
• Memory alignment: Fused kernels may require specific tensor layouts.
• Hardware support: The target NPU must have resources for the combined operation.
• Register pressure: Fusion increases register usage per thread, which can limit occupancy. The compiler must perform a cost-benefit analysis, sometimes choosing partial fusion or a different graph compilation strategy to balance these factors. Excessive fusion can lead to register spilling, which harms performance.

The foundational compiler process that transforms a high-level neural network computational graph into an optimized, hardware-specific sequence of low-level operations. This pipeline applies critical transformations, including operator fusion, layout changes, memory allocation, and instruction scheduling, to generate efficient executable code for a target NPU or GPU.

An automated, empirical optimization process that searches for the optimal configuration parameters for a computational kernel on specific hardware. It explores a vast space of parameters—such as thread block size, loop unrolling factors, and tile dimensions—by running performance benchmarks to find the configuration that delivers the highest throughput or lowest latency, crucial for vendor-specific NPUs.

The compiler stage where a high-level, framework-specific operation (e.g., a PyTorch nn.Conv2d layer) is decomposed or 'lowered' into a sequence of primitive, hardware-agnostic operations. This creates the intermediate representation that subsequent passes, like fusion and scheduling, operate on. Lowering may map directly to vendor-specific intrinsics for maximum performance.

A critical compiler optimization for managing memory hierarchy. It partitions loop iterations into smaller blocks or tiles to fit data into fast, but limited, cache memory (L1/L2). This drastically improves data locality and reuse, reducing expensive accesses to slower global memory. Effective tiling is essential for achieving peak performance on NPUs with complex cache structures.

The strategic orchestration of data movement across different levels of memory in an accelerator (e.g., DRAM → SRAM → registers). Techniques include:

• Software-managed caches explicitly controlled by the compiler.
• Prefetching data before it's needed.
• Minimizing bank conflicts in shared memory. This discipline is the complement to fusion; while fusion reduces total data movement, hierarchy management optimizes the movement that remains.

An analytical performance model used to diagnose hardware utilization bottlenecks. It plots attainable performance (FLOPS/s) against operational intensity (Ops/Byte). The model reveals whether a kernel (fused or unfused) is compute-bound (limited by peak FLOPS) or memory-bound (limited by memory bandwidth). Fusion directly increases operational intensity, moving kernels rightward on the roofline plot toward compute-bound performance.