Fused Conv-BN-ReLU

Each individual GPU kernel launch incurs significant fixed latency for scheduling and setup. Fusing three separate launches (Conv, BN, ReLU) into one eliminates two-thirds of this overhead. This is critical in inference where latency is paramount and batch sizes can be small, making launch overhead a larger fraction of total time.

• Key Impact: Dramatically improves latency for real-time applications like video processing or autonomous vehicle perception.
• Hardware Context: Particularly beneficial on GPUs where launch latency is non-trivial, and for models with many sequential small operators.

The canonical unfused pattern requires writing the large convolution output tensor to global GPU memory (DRAM), then reading it back for batch normalization, writing the result again, and finally reading it for ReLU. This creates a memory-bound bottleneck.

Fusion keeps intermediate tensors in fast on-chip memory (registers or shared memory), performing all computations in a single pass. This:

• Eliminates intermediate stores/loads to high-latency DRAM.
• Increases arithmetic intensity (FLOPs per byte of DRAM access), moving the workload closer to being compute-bound.
• Reduces total memory bandwidth pressure, a key constraint in data center inference servers.

Fusing allows the compiler to apply algebraic optimizations that are impossible across separate kernels. For Conv-BN-ReLU, the batch normalization operation (scale, shift, mean, variance) can be folded directly into the convolution's weights and bias during compilation or at model load time.

This transforms the sequence into a single, modified convolution followed by ReLU: ReLU(BN(Conv(x))) → ReLU(Conv'(x) + b')

• Result: The computationally expensive per-channel mean/variance calculations are removed from the runtime critical path.
• Benefit: The fused kernel executes only the essential convolution and pointwise ReLU, delivering the mathematical equivalent with fewer operations.

Modern GPU architectures have complex memory hierarchies (L1/L2 cache, shared memory). The unfused pattern causes the large intermediate activation tensor to evict useful data from cache after each operation, as it moves through the memory system.

A fused kernel operates on tiles of data, performing all three operations on a tile while it remains in the fastest levels of cache. This principle of fusion for cache maximizes data reuse and minimizes costly cache misses.

• Pattern: Load input tile → Compute Conv → Immediately normalize → Immediately apply ReLU → Store output tile.
• Outcome: Predictable, efficient memory access patterns that hardware prefetchers can optimize for.

Eliminating intermediate storage reduces the peak memory required during the layer's execution. This is crucial for:

• Deployment on edge devices with tightly constrained RAM (e.g., mobile phones, embedded systems).
• Enabling larger batch sizes on a fixed memory budget in data center GPUs, improving throughput.
• Co-locating more model components in GPU memory to avoid slow PCIe transfers.

Quantitative Example: For a 256x256 feature map with 512 channels (FP16), a single intermediate tensor is ~128 MB. Removing two such intermediates saves ~256 MB of transient memory pressure per layer.

A fused Conv-BN-ReLU operator is treated as a single node in the computational graph. This simplifies the graph topology, enabling more aggressive downstream optimizations by compilers like XLA, TVM, or Torch Inductor.

• Larger Fusion Groups: The fused op can itself become a candidate for vertical fusion with preceding or following layers (e.g., with a pooling layer).
• Improved Scheduling: The compiler has fewer, coarser-grained nodes to schedule, leading to more efficient parallel execution plans.
• Static Planning: With a known, stable fused pattern, ahead-of-time (AOT) compilers can generate highly tuned, platform-specific kernels during model compilation, rather than at runtime.

Kernel fusion is a low-level compiler optimization that merges multiple GPU or accelerator kernels into a single, unified kernel. The primary goals are to:

• Reduce kernel launch overhead by amortizing the fixed cost of dispatching work to the hardware.
• Improve data locality by keeping intermediate results in fast registers or shared memory instead of writing them to slow global memory.
• Increase arithmetic intensity by chaining computations before writing final outputs. This is the hardware-specific implementation of the higher-level operator fusion concept.

Operator fusion is a graph-level optimization performed on a neural network's computational graph. Adjacent nodes (operators) are merged into a single, compound operation before kernel code is generated. This is critical because:

• It minimizes intermediate memory allocations and transfers between layers.
• It exposes optimization opportunities (like loop fusion) that are invisible across separate operators.
• It simplifies the graph, reducing scheduling complexity for the runtime. Fused Conv-BN-ReLU is a direct result of applying operator fusion to a Convolution, BatchNorm, and ReLU subgraph.

Graph fusion is the automated process of identifying and merging subgraphs within a computational graph. Unlike manual pattern matching, it uses algorithms to:

• Traverse the dataflow graph to find fusible operator sequences.
• Apply fusion heuristics or a cost model to decide if fusion is profitable.
• Create fusion groups that are passed to a fusion compiler for kernel generation. Frameworks like XLA, TVM, and MLIR implement sophisticated graph fusion passes that automatically discover patterns like Conv-BN-ReLU and many others.

A fusion compiler is a specialized compiler backend responsible for generating high-performance fused kernels from an intermediate representation (IR).

• XLA (Accelerated Linear Algebra): Google's compiler for TensorFlow/JAX. It performs aggressive fusion, including horizontal and vertical fusion, and is a primary driver for fused ops in production.
• TVM (Apache TVM): A deep learning compiler stack that uses its scheduling language (AutoTVM, Ansor) to generate optimized fused kernels for diverse hardware backends.
• MLIR (Multi-Level IR): A compiler infrastructure that provides dialects (e.g., Linalg, Affine) and transformation passes to represent and perform fusion optimizations at multiple abstraction levels.

Deciding what to fuse is a complex optimization problem. Compilers use:

• Fusion Heuristics: Rule-based decisions (e.g., "always fuse elementwise ops with their producer").
• Cost Model for Fusion: A predictive model that estimates the runtime of fused vs. unfused versions, considering:
  • Memory-bound vs. Compute-bound characteristics.
  • Kernel launch overhead savings.
  • Register pressure and cache utilization changes. Fusion profitability analysis uses these models to ensure fusion provides a net performance gain and doesn't cause resource contention.

These are two fundamental patterns for combining operators:

• Vertical Fusion (Producer-Consumer): Merging sequentially dependent ops. Example: Fusing a Convolution (producer) with a following ReLU (consumer). This is the pattern used in Conv-BN-ReLU. It reduces intermediate memory traffic.
• Horizontal Fusion (Sibling Fusion): Merging independent ops that consume the same input or run in parallel. Example: Fusing two separate ReLU operations applied to different tensors. This reduces kernel launch overhead by executing parallel work in a single launch. Advanced compilers perform both types to maximize hardware utilization.