Roofline Model

Operational Intensity is the primary x-axis metric in the Roofline Model, defined as the number of floating-point operations (FLOPs) performed per byte of data transferred between the processor and main memory (DRAM). It is calculated as:

• Formula: Operational Intensity = Total FLOPs / Total DRAM Bytes Transferred
• High Intensity: Kernels with high operational intensity (e.g., dense matrix multiplication) are compute-bound.
• Low Intensity: Kernels with low operational intensity (e.g., element-wise vector addition) are memory-bound. This metric classifies an application's fundamental bottleneck.

Attainable Performance is the primary y-axis metric, measured in GigaFLOPs per second (GFLOPS) or TeraFLOPs per second (TFLOPS). It represents the actual computational throughput achieved by a kernel. In the Roofline plot:

• A point's vertical position shows its measured performance.
• The roofline ceiling above it shows the maximum possible performance for its operational intensity.
• The gap between the point and the ceiling indicates performance optimization headroom. The goal of kernel fusion and other optimizations is to move points vertically toward the roofline.

The Ridge Point is the critical operational intensity value where the performance limit transitions from being constrained by memory bandwidth to being constrained by peak computational throughput. It is calculated as:

• Formula: Ridge Point = Peak Memory Bandwidth (GB/s) / Peak Compute Performance (GFLOPS/byte)
• Left of Ridge: Applications are memory-bound; performance scales linearly with operational intensity.
• Right of Ridge: Applications are compute-bound; performance is capped at the hardware's peak FLOP/s rate. This point defines the two fundamental regimes of the model.

The Memory-Bound Ceiling is the diagonal line on the left side of the roofline plot. It represents the maximum performance achievable when an application is limited by the system's memory bandwidth.

• Slope = Bandwidth: The slope of this line equals the system's peak memory bandwidth (in units of GFLOPS per byte).
• Governing Law: Performance ≤ Operational Intensity × Peak Memory Bandwidth.
• Optimization Target: For kernels under this ceiling, optimizations must focus on reducing DRAM traffic (e.g., via kernel fusion, tiling, improving memory coalescing) to increase effective operational intensity and move the point to the right.

The Compute-Bound Ceiling is the horizontal line on the right side of the roofline plot. It represents the maximum performance achievable when an application is limited by the processor's raw computational capabilities.

• Flat Line: Performance is capped at the hardware's peak FLOP/s rate, regardless of increased operational intensity.
• Governing Law: Performance ≤ Peak Compute Throughput.
• Optimization Target: For kernels under this ceiling, optimizations must focus on improving computational efficiency (e.g., increasing kernel occupancy, leveraging tensor cores, maximizing instruction-level parallelism) to move the point vertically toward the flat ceiling.

For Neural Processing Unit (NPU) acceleration, the Roofline Model is used to:

• Diagnose Bottlenecks: Identify if a fused kernel or model layer is memory-bound or compute-bound on the target NPU.
• Guide Kernel Fusion: Fusing multiple operations (e.g., Conv + Bias + ReLU) increases operational intensity by keeping intermediate data in fast on-chip memory, moving the kernel's point to the right on the plot, potentially crossing the ridge point into the compute-bound region.
• Evaluate Hardware Fit: Compare the rooflines of different NPUs to select the best architecture for a specific AI workload's operational intensity profile.
• Set Optimization Goals: Quantify the performance gap to the ceiling, providing a clear target for compiler engineers and performance architects.

Arithmetic Intensity is the fundamental metric that determines an application's position on the Roofline Model's x-axis. It is defined as the number of floating-point operations (FLOPs) performed per byte of data transferred between the processor and DRAM. This ratio classifies workloads:

• Compute-Bound: High arithmetic intensity. Performance is limited by the processor's peak FLOP/s.
• Memory-Bound: Low arithmetic intensity. Performance is limited by the available memory bandwidth. Optimizations like kernel fusion and loop tiling directly increase arithmetic intensity by reducing off-chip data movement.

Kernel Fusion is a critical compiler optimization that merges multiple, separate computational kernels into a single, larger kernel. This directly impacts the Roofline Model by:

• Reducing Global Memory Traffic: Eliminates writes and subsequent reads of intermediate results to/from DRAM.
• Increasing Arithmetic Intensity: More operations are performed per byte fetched from memory, potentially moving the workload rightward on the Roofline graph.
• Reducing Launch Overhead: Minimizes latency from repeated kernel invocations. For example, fusing a convolution, bias add, and ReLU activation into one kernel is a standard optimization in deep learning compilers like TVM or XLA.

Memory Bandwidth is the hardware ceiling that forms the sloping 'roof' in the Roofline Model. It represents the maximum rate (in GB/s) at which data can be read from or written to main memory. The attainable performance for a memory-bound kernel is calculated as Performance ≤ Arithmetic Intensity × Peak Bandwidth. Key factors include:

• Memory Hierarchy: Effective bandwidth to caches (L1, L2) is orders of magnitude higher than to DRAM.
• Memory Coalescing: An optimization where concurrent memory accesses from threads in a warp are combined into a single transaction, maximizing effective bandwidth utilization.
• Hardware-Specific: Peak bandwidth is a fixed property of the accelerator (e.g., NVIDIA H100: ~3.35 TB/s).

Peak Compute Throughput is the horizontal ceiling in the Roofline Model, representing the maximum FLOP/s the hardware can sustain. This is the limiting factor for compute-bound applications. It is determined by:

• Clock Frequency and Core Count of the processor.
• Specialized Units: The presence of Tensor Cores or Matrix Engines that provide vastly higher throughput for specific operations (e.g., FP16/BF16 matrix multiplies).
• Numerical Precision: Peak throughput is typically quoted for a specific precision (e.g., FP32, FP16, INT8). A kernel must use the hardware's most efficient data types and instructions to approach this theoretical peak.

Loop Tiling (or blocking) is a loop transformation that partitions a large iteration space into smaller blocks or 'tiles'. This optimization is essential for moving data efficiently within the memory hierarchy, directly influencing a kernel's effective arithmetic intensity.

• Principle: Size tiles to fit the working set of data into a faster, smaller memory (e.g., shared memory on a GPU or cache on a CPU).
• Impact: Dramatically reduces accesses to slow global memory, increasing the operational intensity within the tile.
• Roofline Context: Successful tiling can shift a kernel's performance from the memory-bound slope to beneath the compute-bound ceiling on the model.

Auto-Tuning is an empirical, search-based optimization process used to find the best-performing configuration for a kernel on specific hardware. It directly operationalizes the Roofline Model's insights.

• Search Space: Explores parameters like tile sizes, loop unroll factors, vector widths, and thread block dimensions.
• Objective: Maximize performance by minimizing execution time, which on the Roofline Model means pushing the kernel's performance point as high as possible under the relevant roof.
• Frameworks: Tools like Ansor (TVM) and Triton use auto-tuning to automatically generate high-performance kernels, navigating the trade-offs between memory bandwidth and compute utilization implied by the Roofline analysis.