Operator Fusion

The primary benefit of fusion is the elimination of intermediate tensor storage. In a standard sequence like Convolution → BatchNorm → ReLU, each layer writes its full output tensor to RAM before the next layer reads it. Fusion computes the final result in a single pass, reusing the input buffer for the output. This in-place computation can reduce the activation memory (RAM footprint) by 50% or more for deep networks, which is often the limiting factor for model deployment on MCUs.

Reading and writing tensors from SRAM is a major energy consumer on microcontrollers. Each unfused layer requires:

• Reading input tensors from memory.
• Writing output tensors back to memory.
  Fused kernels keep intermediate values in CPU registers or local cache, drastically cutting the number of costly off-chip memory accesses. This reduces power consumption and is critical for battery-operated devices where memory I/O can dominate energy use.

Fused operations exhibit excellent data locality. The data needed for the combined computation is loaded once and reused extensively across the fused sub-operations within the kernel. This maximizes the utility of the microcontroller's small, fast cache memory (if present) and minimizes stalls caused by waiting for data from slower main memory. Techniques like loop tiling are often combined with fusion to further optimize this locality.

Executing each layer as a separate kernel involves function call overhead, argument passing, and potential setup/teardown for hardware accelerators. On a resource-constrained MCU, this overhead can be significant relative to the actual computation time. Fusing multiple operations into one kernel amortizes this fixed cost over more work, leading to higher computational efficiency and lower overall latency.

Fusion allows the compiler to statically schedule and pre-compute parts of the graph. A classic example is fusing a Batch Normalization layer into a preceding Convolution. The BN operation (scale, shift) can be folded into the convolution's weights and bias at compile-time, resulting in a mathematically equivalent but faster single convolution with zero runtime cost for BN. This also applies to scaling operations from quantization.

A fused operation presents a larger, more complex computational pattern to the compiler backend. This allows for the use of more advanced SIMD instructions and processor-specific optimizations that would not be possible on the smaller, individual operations. Frameworks like CMSIS-NN extensively use hand-optimized, fused kernels (e.g., convolution with ReLU) to fully exploit the capabilities of Arm Cortex-M series microcontrollers.

Operator fusion works by analyzing the compute graph of a neural network. The compiler identifies chains of operations where the output of one layer is the immediate input to the next, with no external dependency. Common fusion patterns include:

• Convolution + BatchNorm + Activation: The most impactful fusion, combining linear, scaling, and non-linear steps.
• Element-wise Operations: Sequences like Add + ReLU can be merged into a single pass over the data.

The fused kernel computes the combined mathematical transformation in one loop, writing the final result directly to the output buffer. This bypasses the need to store the intermediate tensor from the first operation, which is the primary source of RAM footprint reduction.

The primary advantage of fusion is the drastic reduction in memory bandwidth and latency, which are the dominant constraints on microcontrollers.

Key Impacts:

• Peak RAM Reduction: By eliminating intermediate buffers, fusion can reduce the activation memory requirement by 30-50% for common layer patterns.
• Fewer Memory Accesses: Each memory read/write consumes power and time. Fusion minimizes transfers between the processor and SRAM.
• Improved Cache Locality: The fused computation keeps data in CPU registers or cache across what were previously separate operations, avoiding costly cache misses.

For example, fusing a Conv2D layer with a subsequent ReLU activation avoids writing the full conv output tensor to RAM only to immediately read it back for the ReLU.

Major frameworks for microcontroller deployment implement fusion as a graph-level optimization pass.

TensorFlow Lite Micro (TFLM): Uses a MicroOpResolver and a graph optimizer that applies fusion patterns defined in the TFLite converter (e.g., FuseConv2dAndActivation). The fused operation is implemented as a specialized kernel.

CMSIS-NN: Provides hand-optimized, fused kernels like arm_convolve_s8 (convolution) and arm_depthwise_separable_conv_3x3_nonsquare_s8 which inherently include activation functions.

TVM / Apache TVM Micro: Its Relay intermediate representation and compiler stack perform explicit operator fusion passes, searching for fusible patterns and generating combined C code for the target device.

The feasibility and benefit of fusion depend on the target microcontroller's architecture.

Constraints & Considerations:

• Register Pressure: A fused kernel performs more work before writing results, potentially requiring more CPU registers. If registers spill to RAM, benefits are lost.
• Supported Data Types: Fusion must respect the quantization scheme. Fusing INT8 convolution with an INT16 batch norm requires careful handling of intermediate precision.
• Memory Alignment: Fused kernels must ensure tensor data aligns with the requirements of SIMD instructions for optimal performance.

Thus, fusion is not always automatic; it requires the compiler to model hardware resources to decide when fusion is profitable.

Operator fusion is often confused with in-place computation, but they are distinct, complementary optimizations.

A system can use both: a fused Conv+ReLU kernel can also be scheduled to execute in-place, overwriting its input buffer. This combination delivers the maximum reduction in both compute latency and RAM footprint.

Consider a standard block: Conv2D -> BatchNorm -> ReLU. Without fusion, this requires three kernel launches and two intermediate tensors.

Fused Transformation: The fused kernel applies the combined formula in one pass: Output = ReLU( BN( Conv(Input) ) ) Which, after folding BatchNorm's scale (γ) and shift (β) into the convolution weights and bias, simplifies to: Output = ReLU( (Conv(Input) * γ / sqrt(σ²+ε)) + (β - μ * γ / sqrt(σ²+ε)) )

The compiler pre-computes the fused weight (W_fused = W * γ / sqrt(σ²+ε)) and fused bias (b_fused = (b * γ / sqrt(σ²+ε)) + (β - μ * γ / sqrt(σ²+ε))). The kernel then computes: Output = ReLU( Conv_fused(Input) ). This single operation is far more efficient than the three-step sequence.

A compute graph is a directed acyclic graph (DAG) representation of a neural network where nodes represent operations (ops) and edges represent the multidimensional data arrays (tensors) flowing between them. This abstract representation is the fundamental data structure that enables compiler optimizations like operator fusion.

• The graph is analyzed to identify sequences of operations that can be merged.
• It allows for static scheduling, where execution order and memory allocation are determined at compile-time for deterministic runtime.

Kernel optimization refers to the manual or automated low-level tuning of fundamental neural network operation implementations for a specific microcontroller architecture. While operator fusion creates a new, combined kernel, each individual kernel must be highly optimized.

• Techniques include leveraging SIMD instructions for parallel data processing.
• Loop unrolling and loop tiling are used to reduce overhead and improve data cache locality.
• Libraries like CMSIS-NN provide a set of these hand-optimized kernels for Arm Cortex-M processors.

Static memory allocation is a memory management strategy where all buffers for model weights, activations, and intermediate tensors are pre-allocated at compile-time. This is essential for fusion's benefits.

• Eliminates runtime allocation overhead and memory fragmentation.
• Allows the compiler to precisely plan in-place computation, where a fused kernel's output can overwrite its input buffer.
• Directly reduces the RAM footprint, which is often the primary constraint on microcontrollers.

In-place computation is an optimization where the output of a neural network layer is written directly into the memory location of its input, reusing memory buffers. Operator fusion enables more aggressive in-place strategies.

• A fused Conv-BatchNorm-ReLU operation can compute the final activation without writing intermediate tensors to RAM.
• This drastically reduces the peak RAM usage during inference, often the limiting factor for model deployment on MCUs.
• Requires careful dependency analysis within the compute graph to ensure data correctness.

Integer quantization is a model compression technique that constrains model parameters and activations to integer values (e.g., INT8). It synergizes powerfully with operator fusion.

• Fused kernels can be implemented using integer-only arithmetic, avoiding costly floating-point emulation on MCUs.
• The combined operation executes as a single, quantized kernel, maintaining numerical consistency and efficiency.
• Techniques like quantization-aware training (QAT) help maintain accuracy for fused, quantized operators.