Operator Fusion

In unfused graphs, each operator writes its output tensor to off-chip DRAM and the next operator reads it back. Operator fusion collapses this chain, keeping intermediate activations in on-chip buffers or registers.

• Eliminates costly round-trips to external memory
• Critical for FPGA deployments where DRAM bandwidth is the primary bottleneck
• Enables streaming architectures that process data as a continuous flow

For a typical CNN classifier, fusing convolution with batch normalization and ReLU can reduce external memory traffic by up to 3x per fused block.

Each discrete GPU kernel or FPGA accelerator invocation incurs a fixed scheduling cost—dispatching work, configuring registers, and synchronizing pipelines. Operator fusion amortizes this overhead across multiple operations.

• Reduces kernel launch latency from microseconds to zero for fused operations
• Critical for real-time RF inference where sub-millisecond pipelines are required
• Minimizes host-to-accelerator synchronization points

In GPU inference, fusing element-wise operations (bias, scale, activation) can eliminate dozens of kernel launches per layer, saving 10-50 microseconds per fused block.

The roofline model reveals that many neural network layers are memory-bound rather than compute-bound. Operator fusion increases the ratio of arithmetic operations per byte loaded from memory.

• Transforms memory-bound operations into compute-bound kernels
• Better utilizes DSP slices and systolic arrays on FPGA fabric
• Enables higher sustained throughput on fixed hardware

For example, fusing a depthwise convolution with its subsequent pointwise projection increases arithmetic intensity by keeping the intermediate feature map entirely in local registers.

Several fusion patterns are particularly effective for modulation recognition models deployed on edge hardware:

• Conv + BatchNorm + ReLU: Mathematically absorb BN parameters into convolution weights, then fuse activation
• Conv + Bias + Activation: Merge additive bias and non-linearity into a single kernel
• Element-wise chain fusion: Combine sequential Add, Mul, and activation operations
• Horizontal fusion: Merge parallel branches with identical input tensors

Tools like TensorRT and Vitis AI automatically identify and apply these patterns during model compilation for GPU and FPGA targets respectively.

Operator fusion is a prerequisite for streaming architectures on FPGAs, where data flows continuously through a deep pipeline without intermediate buffering.

• Fused operators form pipeline stages that process IQ samples as they arrive
• Eliminates the need for ping-pong buffers between every layer
• Achieves deterministic, ultra-low latency suitable for real-time spectrum monitoring

A fully fused modulation classifier can begin producing classification results within microseconds of receiving the final IQ sample, rather than waiting for complete layer-by-layer execution.

Operator fusion and integer-only inference are complementary optimizations. Fusion reduces the number of quantization boundaries where precision must be preserved.

• Fewer quantization nodes mean less accumulated rounding error
• Fused integer kernels avoid float-to-int conversions between operations
• Enables cross-layer equalization to be applied across fused blocks

When combined with quantization-aware training, fused integer kernels can match floating-point accuracy while achieving 4x throughput improvements on FPGA DPU cores.

A pre-deployment optimization that mathematically absorbs batch normalization parameters into the preceding convolutional layer's weights and biases. During inference, batch normalization becomes a fixed linear transformation: y = γ(x - μ)/σ + β. By folding these parameters into the convolution weights (W_folded = W * γ/σ) and biases (b_folded = (b - μ) * γ/σ + β), the separate normalization kernel is eliminated entirely. This reduces kernel launch overhead and eliminates intermediate memory writes, making it a critical precursor step before broader operator fusion passes.

An FPGA dataflow design pattern where fused operator chains are implemented as deep pipelines processing continuous IQ sample streams. Unlike batch processing, streaming architectures eliminate external memory round-trips between fused operations: intermediate activations flow directly through FIFO buffers between pipeline stages. For real-time modulation classification, this enables deterministic latency as low as single-digit microseconds, with the fused convolution-activation-pooling chain operating as a single monolithic hardware module.

The fundamental arithmetic primitive that operator fusion optimizes for. A MAC operation computes a ← a + (b × c) in a single cycle on DSP48 slices. When convolution, bias addition, and activation functions are fused, the output of one MAC stage feeds directly into the next without register spilling. Modern FPGA DSP tiles can perform INT8 MAC operations at rates exceeding 2 TOPS, and fusion ensures these units remain continuously utilized rather than stalling between separate kernel invocations.

A visual performance analysis tool that determines whether a fused operator chain is compute-bound or memory-bound. The model plots achievable throughput (FLOPs/sec) against operational intensity (FLOPs/byte). Operator fusion increases operational intensity by reducing intermediate memory traffic—moving workloads rightward on the roof-line curve. For modulation classifiers on FPGA, fusing depthwise separable convolutions can shift a layer from the memory-bound sloped region to the compute-bound flat ceiling, unlocking the full DSP capacity.

A complementary optimization that maximizes the bandwidth benefits of operator fusion. Multiple low-precision activations (e.g., four INT8 values) are packed into a single wide memory word (e.g., 32-bit). When consecutive fused operators consume and produce packed data, the SIMD parallelism is preserved across kernel boundaries without unpacking overhead. In FPGA-based modulation classifiers, packing four INT8 IQ samples per word allows fused convolution-bias-ReLU chains to process 4x more data per memory transaction.