Operator Fusion

The primary motivation for operator fusion is to eliminate intermediate tensor reads and writes to off-chip DRAM. In a non-fused graph, each operator loads its input from memory and writes its output back. Fusion collapses this chain, keeping intermediate data in on-chip registers or shared memory (SRAM). This directly addresses the memory wall bottleneck, where data movement consumes far more energy and time than the actual computation, a critical constraint for battery-operated medical wearables.

Compiler frameworks like Apache TVM and XLA target specific fusion motifs:

• Conv-BatchNorm-ReLU: The canonical pattern for computer vision models, fusing convolution, normalization, and activation into a single kernel.
• MatMul-Bias-Add: Fuses matrix multiplication with a subsequent bias addition, common in transformer attention layers.
• Element-wise Chains: Sequences of pointwise operations like Add, Mul, and activation functions are merged to avoid per-element memory traffic.
• Injector Pattern: A lightweight operation, like a bias add, is 'injected' into the output stage of a compute-heavy preceding operator.

Fusion strategies are categorized by their direction in the computational graph:

• Vertical Fusion: Merges operators in a sequential producer-consumer relationship along a single data path. This is the most common form, directly reducing intermediate data materialization.
• Horizontal Fusion: Combines parallel branches that operate on the same input tensor, such as the separate Key, Query, and Value projections in a multi-head attention block, into a single batched matrix multiplication for higher arithmetic intensity.

Fusion is not universally beneficial. A compiler must use a machine learning-based cost model to predict the performance of fused vs. unfused implementations on the target NPU or GPU. Over-fusion can increase register pressure, causing register spilling to slower memory and degrading performance. The cost model evaluates trade-offs between reduced memory traffic and increased compute kernel complexity, ensuring the fused kernel fits within the accelerator's local memory budget.

On edge devices like medical imaging tablets, operator fusion can reduce end-to-end inference latency by 20-40% for standard CNN architectures. The primary gain comes from eliminating kernel launch overhead and memory transaction delays. For real-time applications such as streaming inference on digital stethoscopes, this reduction is critical to meeting a strict latency budget, ensuring that a cardiac arrhythmia prediction is generated within the clinically acceptable window.

Operator fusion is most effective when combined with INT8 quantization. A fused Conv-BatchNorm-ReLU kernel can be designed to operate entirely on 8-bit integers, using a single requantization step at the output. Without fusion, each discrete operator would require its own quantization and dequantization steps, introducing numerical precision loss and computational overhead. The fusion compiler must be aware of the quantization scheme to insert requantization nodes at the correct fused kernel boundaries.

A directed acyclic graph (DAG) representing a neural network's computation where nodes are mathematical operations (e.g., Conv2D, ReLU) and edges are tensors flowing between them. Operator fusion rewrites this graph by merging adjacent, compatible nodes into a single fused kernel.

• Static Graphs: Defined and optimized once before execution (e.g., TensorFlow 1.x, ONNX).
• Dynamic Graphs: Defined on-the-fly during execution, making ahead-of-time fusion more challenging.
• Graph IR: Intermediate representations like MLIR or Relay are the levels where fusion pattern-matching occurs.

The primary bottleneck that operator fusion solves. Processor speeds have outpaced memory access speeds, meaning the energy and time cost of loading and storing intermediate tensors to off-chip DRAM often dominates total inference latency.

• A fused kernel keeps intermediate data in on-chip registers or shared memory (SRAM).
• Eliminates costly round-trips to DRAM between trivial operations like Add followed by ReLU.
• Critical for medical wearables where DRAM access can consume orders of magnitude more energy than a multiply-add operation.

Two fundamental strategies for merging operations:

• Vertical Fusion: Merges a producer operation directly with its consumer (e.g., fusing a MatMul with its subsequent BiasAdd and ReLU activation). This is the most common form, creating a single deep kernel.
• Horizontal Fusion: Merges independent operations that execute on the same input tensor (e.g., fusing a Sigmoid and a Tanh branch that both read from the same tensor). This reduces redundant memory reads.
• Modern compilers like Apache TVM apply both strategies in a single optimization pass.

The low-level compiler mechanics that enable operator fusion. Loop fusion collapses the iteration spaces of multiple operations into a single nested loop, while loop tiling partitions this fused loop into blocks that fit entirely in the accelerator's local scratchpad memory.

• Polyhedral compilation is a mathematical framework used to automatically find legal and optimal loop fusion and tiling configurations.
• Without tiling, a fused kernel that exceeds on-chip memory capacity will spill to DRAM, negating the performance benefit.
• This is a key technique for mapping fused operators onto systolic arrays in NPUs.

The fixed cost of dispatching a computational kernel to a GPU or NPU. Each discrete operator in an unfused graph requires a separate kernel launch, involving driver calls, command buffer submission, and synchronization.

• For a graph with hundreds of small, element-wise operations, launch overhead can dominate total execution time.
• Operator fusion amortizes this cost by replacing many small kernel launches with a single, larger one.
• This is especially critical on mobile GPUs and NPUs where driver overhead is significantly higher than on server-grade hardware.