Roofline Model

Operational Intensity is the primary independent variable in the Roofline Model. It is defined as the number of Floating-Point Operations (FLOPs) performed per byte of data transferred between the processor and the main memory hierarchy. It is measured in FLOPs/Byte.

• High Intensity (>10-100 FLOPs/Byte): Kernels like dense matrix multiplication are compute-bound, limited by the chip's peak FLOP/s.
• Low Intensity (<1 FLOP/Byte): Kernels like element-wise addition are memory-bound, limited by the system's memory bandwidth.

Calculating a kernel's operational intensity requires analyzing its algorithm to count total FLOPs and the minimum bytes that must be moved from DRAM.

Peak Compute Performance is the theoretical maximum number of floating-point operations a processor can perform per second, measured in FLOP/s (e.g., TeraFLOP/s or TFLOPS). This forms the horizontal 'roof' of the model.

• This limit is determined by hardware specs: clock frequency, number of cores/ALUs, and SIMD width.
• It is often specified for different numerical precisions (e.g., FP32, FP16, INT8). A modern NPU might have a much higher peak for INT8 than for FP32.
• A kernel's performance cannot exceed this ceiling. If a kernel's operational intensity is high enough that its attainable performance plateaus at this line, it is compute-bound.

Peak Memory Bandwidth is the theoretical maximum rate at which data can be transferred between the processor and its main memory (e.g., HBM, LPDDR), measured in Bytes/s (e.g., GB/s). This forms the diagonal 'slope' of the model.

• This limit is determined by the memory bus width, data rate, and number of memory channels.
• The slope of the roofline is defined as: Attainable GFLOP/s = (Bandwidth in GB/s) * (Operational Intensity in FLOP/Byte).
• A kernel's performance cannot exceed this slope. If a kernel's performance falls along this line, it is memory-bound; every increase in operational intensity directly increases performance.

The Attainable Performance Region is the space below both the horizontal compute roof and the diagonal bandwidth slope. Any real computational kernel will plot as a point within this region.

• Location Diagnoses Bottleneck: A point near the diagonal slope indicates a memory-bound kernel. Optimization should focus on improving data reuse (e.g., via tiling), reducing transfers, or increasing operational intensity.
• A point near the horizontal roof indicates a compute-bound kernel. Optimization should focus on improving instruction-level parallelism, using vector/SIMD instructions, or leveraging specialized functional units (e.g., tensor cores).
• The distance from the roof indicates the optimization headroom available for that kernel on that hardware.

A Kernel Roofline plot places a specific kernel's measured performance (in GFLOP/s) against its calculated operational intensity. This reveals its precise bottleneck.

• Optimization Vector: The vertical distance from the kernel's point to the roofline represents lost performance. The goal is to move the point vertically upward.
• Moving the Point: To improve a memory-bound kernel, you must increase its operational intensity (moving it rightward) via algorithmic changes like loop tiling, kernel fusion, or using cache-friendly data layouts. This moves it up the slope.
• For a compute-bound kernel, you must improve the kernel's computational efficiency to utilize more of the hardware's peak FLOP/s, moving the point vertically upward against the roof.

The Hierarchical Roofline Model extends the basic model by adding multiple 'ceilings' that represent the performance limits of different levels of the memory hierarchy (e.g., L1 cache, L2 cache, HBM).

• Each memory level has its own bandwidth and associated roof slope.
• A kernel's operational intensity can be recalculated for data sourced from each cache level, showing which level is the current bottleneck.
• This is critical for NPU optimization, where managing data movement between global memory, shared memory/SRAM, and register files is paramount. The model guides optimizations like operator fusion (to keep data in fast memory) and explicit memory hierarchy management.

The Roofline Model's primary use is to classify a kernel's performance limit. A kernel plotted below the roofline is memory-bound: its performance is limited by the speed of data movement from memory. A kernel plotted at the roofline is compute-bound: its performance is limited by the hardware's peak arithmetic throughput.

• Example: A dense matrix multiplication with high operational intensity will likely be compute-bound on a GPU. A sparse matrix-vector multiplication with low operational intensity will be memory-bound.
• Actionable Insight: For a memory-bound kernel, optimize data reuse, improve cache locality, or use compression. For a compute-bound kernel, maximize instruction-level parallelism or use lower precision (e.g., FP16).

Algorithm designers use the Roofline Model during early development to ensure new methods are suited to target hardware. By estimating a new algorithm's operational intensity, developers can predict its performance ceiling before implementation.

• Real-World Case: When designing the Flash Attention algorithm, its creators analyzed the standard attention mechanism's low operational intensity (making it memory-bound). Flash Attention was explicitly designed to increase operational intensity by recomputing on-chip, moving its performance plot closer to the compute roof.
• Use in NAS: Hardware-Aware Neural Architecture Search (NAS) can use roofline-derived metrics (attainable GFLOP/s) as a proxy for latency when evaluating candidate model architectures.

Each accelerator has a unique roofline defined by its peak FLOP/s and memory bandwidth. Optimizing a kernel for an NVIDIA A100 is different than for a Google TPU v4 or an Apple Neural Engine. The model forces explicit consideration of these hardware ceilings.

• Example Workflow:
  1. Profile a convolution kernel on the target NPU to get its achieved GFLOP/s.
  2. Calculate its operational intensity (FLOPs/byte).
  3. Plot it on the NPU's roofline chart.
  4. If memory-bound, apply NPU-specific optimizations like operator fusion to reduce DRAM traffic or utilize specialized on-chip memory (e.g., shared memory/SRAM).

System architects and hardware designers use the Roofline Model for comparative analysis and target setting.

• Procurement: Comparing the rooflines of different accelerators (e.g., GPU vs. TPU) for a specific workload profile (e.g., low-intensity recommendation models vs. high-intensity LLM training).
• Hardware Design: Chip designers set targets for balance (FLOP/s per byte/s). A design with extremely high FLOP/s but low bandwidth will only benefit a narrow set of compute-intensive kernels. The model visualizes this trade-off.
• Historical Example: The shift from CPUs to GPUs for AI was fundamentally a move to a hardware roofline with a much higher compute ceiling and a different balance point.

The basic roofline model can be extended to hierarchical systems. A multi-roofline model can depict different performance limits for different levels of the memory hierarchy (e.g., L1 cache, L2 cache, HBM).

• NUMA Systems: On Non-Uniform Memory Access systems, different rooflines can be drawn for local vs. remote memory access. This highlights the severe performance penalty of not managing NUMA affinity.
• Practical Use: Guides data placement and thread pinning strategies to ensure kernels access local memory, keeping them on the higher, local-memory roofline.

Changing numerical precision (e.g., FP32 → FP16 → INT8) directly alters both axes of the Roofline Model.

• Operational Intensity (X-axis): Lower precision (INT8) means each byte transferred from memory represents more operations, effectively shifting the kernel's plot point to the right.
• Peak Compute (Y-axis): Hardware can typically execute many more low-precision operations per second, raising the compute roofline.
• Result: A kernel that is memory-bound in FP32 may become compute-bound in INT8, unlocking higher performance. This provides a quantitative framework for evaluating Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) benefits.

Operational Intensity is the fundamental metric that determines a kernel's position on the Roofline Model's X-axis. It is defined as the number of floating-point operations (FLOPs) performed per byte of data transferred between the processor and main memory (DRAM).

• Compute-Bound Kernels: Have high operational intensity, placing them on the flat, compute-limited roof of the model.
• Memory-Bound Kernels: Have low operational intensity, placing them on the sloped, bandwidth-limited roof. The primary goal of many optimizations is to increase a kernel's operational intensity by improving data reuse.

Kernel Fusion is a critical compiler optimization that merges multiple sequential operations (e.g., Convolution → BatchNorm → ReLU) into a single, monolithic kernel. This directly targets the Roofline Model by:

• Reducing Memory Traffic: Fused operations pass intermediate results through fast registers or shared memory instead of writing to and reading from slow DRAM.
• Increasing Operational Intensity: Fewer bytes transferred for the same number of FLOPs moves the kernel to the right on the Roofline plot.
• Reducing Kernel Launch Overhead: Eliminates the latency of launching multiple small kernels.

Loop Tiling (or blocking) is a transformation that partitions loop iterations into smaller blocks to exploit the memory hierarchy. It is essential for optimizing memory-bound kernels on the Roofline's sloped region.

• Principle: Data is loaded from slow DRAM into a fast, small cache (L1/SRAM). Computations are performed on this 'tile' of data to maximize reuse before eviction.
• Impact: Dramatically reduces the required memory bandwidth, effectively increasing the operational intensity of the kernel. The tile size is a key tuning parameter, often optimized via auto-tuning to match specific cache sizes.

These are the two fundamental performance regimes defined by the Roofline Model.

• Memory-Bound: Performance is limited by the speed (bandwidth) at which data can be moved from memory. Kernels here have low operational intensity. Optimizations focus on reducing data movement (fusion, tiling, improving cache locality).
• Compute-Bound: Performance is limited by the peak floating-point throughput of the hardware. Kernels here have high operational intensity. Optimizations focus on maximizing compute utilization (increasing parallelism, using tensor cores, ensuring sufficient arithmetic density). The Roofline Model's knee is the point where a kernel transitions from one regime to the other.

Roofline Analysis is the practical application of the Roofline Model to profile and optimize real-world code. The process involves:

1. Instrumentation: Measuring the actual FLOPs and bytes transferred for a kernel using hardware performance counters (e.g., via nvprof, rocprof, or perf).
2. Plotting: Placing the measured point (FLOPs/byte, GFLOPs/sec) on the model with the hardware's roof.
3. Diagnosis: Determining the bound (memory or compute) and the performance gap to the optimal roof.
4. Optimization Guidance: The plot directs engineers to the most impactful optimizations based on the kernel's regime.

Arithmetic Intensity is often used synonymously with Operational Intensity. Precisely, it is the ratio of total arithmetic operations to total data movement. In the context of the Roofline Model for Neural Processing Units (NPUs), this concept is extended to account for specialized dataflows.

• Weight Stationary: Keeps filter weights in a fast buffer, optimizing for layers where weights are reused across many input activations (high intensity).
• Output Stationary: Keeps partial outputs locally, optimizing for layers with large filter sizes.
• NPU-Specific: The effective 'intensity' seen by the system depends heavily on how well the compiler schedules data to leverage these stationary buffers, moving the kernel closer to the compute roof.