Vertical Fusion

Vertical fusion is the merging of a producer operator with its immediate consumer operator within a computational graph. This optimization chains operations that have a direct, sequential data dependency, where the output of one operator is the sole input to the next.

• Objective: Eliminate the need to write the producer's result to slow, high-bandwidth memory (HBM) and immediately read it back for the consumer.
• Graph Traversal: Compilers identify these patterns by analyzing the dataflow graph, looking for edges connecting two nodes where the consumer has no other data dependencies.
• Contrast with Horizontal Fusion: Unlike horizontal fusion, which merges parallel operators, vertical fusion explicitly targets the critical path of sequential execution.

The fusion of vertically dependent operators delivers performance gains through two primary mechanisms:

• Reduced Global Memory Traffic: The most significant benefit. Intermediate tensors are kept in fast on-chip memory (GPU registers or shared memory) instead of being written to and read from DRAM. This is critical for memory-bound operations.
• Lowered Kernel Launch Overhead: Executing one fused kernel instead of two separate kernels amortizes the fixed launch latency associated with each GPU kernel invocation. This overhead becomes substantial in models with many small, sequential operations.

Example Impact: Fusing a GeLU activation directly after a Linear layer can eliminate a full read/write cycle of a potentially large activation tensor, directly translating to higher throughput and lower latency.

Compilers use pattern matching to identify common, profitable sequences for vertical fusion. Well-known examples include:

• Conv-BN-ReLU: The foundational block of CNNs, fusing Convolution, Batch Normalization, and ReLU activation.
• Linear-GeLU / Linear-GELU-Dropout: Common in transformer feed-forward networks.
• LayerNorm-RMSNorm-Silu: Common normalization and activation sequences in modern architectures.
• Elementwise Chains: Sequences like Add -> ReLU -> Multiply where all ops are pointwise.

Frameworks like PyTorch's Inductor and XLA have built-in pattern matchers that automatically rewrite these subgraphs into calls to pre-written or generated fused kernels for optimal performance.

Vertical fusion is implemented within graph-level compilers and just-in-time (JIT) compilation systems. The process involves:

1. Graph Lowering: The high-level model graph is lowered to a mid-level IR (like MLIR's Linalg dialect or Torch IR) consisting of primitive operators.
2. Pattern Matching & Planning: A fusion planner identifies producer-consumer pairs and groups them into fusion groups based on fusion heuristics and a cost model.
3. Kernel Code Generation: The compiler generates a single fused kernel (e.g., CUDA or Metal code) that implements the combined logic of the group.
4. Scheduling: The compiler performs fusion-aware scheduling, deciding on loop structures, tiling, and memory promotion (e.g., to shared memory) for the fused kernel.

Key compilers performing this optimization include XLA (for JAX/TensorFlow), Apache TVM, MLIR-based compilers, and PyTorch's torch.compile with its Inductor backend.

Not all vertical fusions are beneficial. A fusion planner must analyze fusion profitability based on:

• Memory vs. Compute Bound: Fusing a memory-bound operator (e.g., activation) with a compute-bound one (e.g., matmul) is almost always profitable. Fusing two compute-heavy ops may offer less benefit.
• On-Chip Memory Pressure: Fusing too many operations can exhaust GPU registers or shared memory, leading to register spilling to slower memory, which harms performance.
• Parallelism Reduction: Fusing may limit independent scheduling opportunities for the original operators.
• Kernel Complexity: Extremely large fused kernels can hinder GPU warp scheduler efficiency and increase compilation time.

Compilers use a cost model for fusion to estimate execution cycles with and without fusion, often deciding to fuse only when a significant reduction in DRAM traffic is predicted.

While not fusion in the traditional sense, CUDA Graphs address a similar high-level goal: reducing launch overhead for sequences of operations. A CUDA Graph captures a whole series of kernel launches and memory copies into a single, replayable unit.

• Coarse-Grained Optimization: It reduces CPU overhead and GPU driver latency by submitting the entire graph once, rather than each kernel individually.
• Complementary to Fusion: CUDA Graphs are often used after aggressive kernel fusion. The graph contains fewer, larger fused kernels, making graph capture and replay even more efficient.
• Use Case: Essential for achieving peak throughput in inference servers using frameworks like NVIDIA Triton, where the same graph of fused kernels is executed repeatedly for different requests.

This is the quintessential example of vertical fusion in convolutional neural networks (CNNs). It combines three sequentially dependent layers:

• Convolution: Applies filters to input feature maps.
• Batch Normalization: Normalizes the output activations.
• ReLU Activation: Applies a non-linear rectification.

Fusion Benefit: The intermediate NCHW tensors between these layers never materialize in global GPU memory. The normalization and activation are computed on-the-fly as the convolution outputs are produced, drastically reducing memory bandwidth pressure. This pattern is so common it is often implemented as a single, hand-tuned kernel in deep learning libraries.

The transformer attention mechanism is a deep sequence of operations: linear projections, scoring, masking, softmax, and aggregation. A naive implementation launches separate kernels for each, writing massive intermediate attention score matrices (O(N²)) to slow HBM.

Vertical Fusion in FlashAttention: The algorithm fuses the entire attention block into a single kernel. It performs the softmax normalization using a online, numerically stable method without materializing the full score matrix. The key innovation is IO-awareness: it tiles computations to keep working data in fast SRAM, recomputing scores as needed. This reduces HBM reads/writes from quadratic to linear in sequence length.

Chains of pointwise operations are prime candidates for vertical fusion. Common patterns include:

• Activation Sequences: GELU → Dropout → Residual Add
• Normalization & Scaling: LayerNorm → Scale → Bias
• Precision Conversion: Dequantize → FP16 Operation → Quantize

Fusion Mechanism: The compiler identifies that these operations are elementwise independent—each output element depends only on the corresponding input element(s). It then generates a single kernel that loops over tensor elements once, applying the entire sequence of transformations in-register. This eliminates multiple passes over the data and associated kernel launch latency and memory bandwidth consumption.

In transformer feed-forward networks and MLP blocks, a linear (fully-connected) layer is immediately followed by a Gaussian Error Linear Unit (GeLU) or Sigmoid Linear Unit (SiLU) activation.

Vertical Fusion Process: The fused kernel computes the matrix multiplication for a block of the output. Instead of writing the block result back to global memory, it immediately applies the non-linear GeLU function (x * Φ(x)) using efficient polynomial approximations. This keeps the intermediate results in registers or shared memory, avoiding a round-trip to main memory. This optimization is critical in memory-bound workloads where the linear layer's output size is large.

Vertical fusion is applied to sequences where a reduction (e.g., sum, max) feeds directly into another operation. A classic example is the fusion of Layer Normalization components.

LayerNorm Fusion Breakdown:

1. Mean Calculation: A reduction kernel sums elements along a dimension.
2. Variance Calculation: Another reduction kernel sums squared differences.
3. Normalize & Scale: Elementwise normalization using the computed mean/variance.

Fused Implementation: A single kernel performs both reductions in one pass over the data using efficient parallel reduction algorithms, computes the statistics, and immediately normalizes the values. This avoids three separate kernel launches and two passes over the input tensor, providing significant latency savings for this common operation.

Operator fusion is the graph-level optimization that merges adjacent computational operators in a neural network's computational graph into a single, compound operation. It is the high-level strategy of which vertical fusion is a specific type.

• Goal: Minimize intermediate memory transfers between operators.
• Scope: Works on the abstract computational graph before low-level kernel generation.
• Example: Deciding to merge a matrix multiplication with a subsequent bias addition and ReLU activation.

Kernel fusion is the low-level compiler technique that combines the GPU or accelerator kernels for multiple operations into a single, unified kernel. It implements the fusion decision at the hardware instruction level.

• Goal: Reduce kernel launch overhead and improve data locality within GPU registers and caches.
• Mechanism: Generates a custom kernel that performs the work of several primitive ops without writing intermediate results to global memory.
• Relationship: Operator fusion often, but not always, leads to kernel fusion. A fused operator is executed by a fused kernel.

Horizontal fusion is the merging of multiple independent operators that consume the same input tensor or operate in parallel on different data within the computational graph. It contrasts with the sequential dependency of vertical fusion.

• Pattern: Multiple branches from a single producer are merged into a single kernel.
• Benefit: Amortizes kernel launch overhead and can improve memory bandwidth utilization by processing multiple outputs in one pass.
• Challenge: Often more complex than vertical fusion due to potentially divergent computation paths.

Fused Multi-Head Attention is a canonical, high-impact example of vertical fusion. It implements the entire attention mechanism—query/key/value projection, scoring, softmax, and aggregation—in a single, highly optimized kernel.

• FlashAttention: A seminal IO-aware fused algorithm that minimizes memory reads/writes by recomputing attention scores on-chip (SRAM), avoiding materializing the large attention matrix to HBM.
• Impact: Enables faster training and inference and allows transformers to process much longer context lengths by reducing memory complexity from O(N²) to O(N).

Fusion profitability is the critical analysis of whether fusing a set of operators yields a net performance gain. It is guided by a compiler's cost model for fusion.

Factors Considered:

• Reduction in Kernel Launch Overhead: Saved latency from fewer GPU launches.
• Improved Data Locality: Reduced traffic to slow global memory.
• Increased Register Pressure: Fusing may require more on-chip registers, which can limit parallelism.
• Decreased Parallelism: A single large kernel may expose less parallel work than several smaller ones.

A fusion planner uses this cost model to explore the search space of possible fusions.