Roofline Model

Operational Intensity is the primary x-axis of the Roofline Model, measured in FLOPs per Byte. It quantifies the amount of useful computational work performed per byte of data transferred from main memory.

• Definition: Operational Intensity = Total FLOPs / Total Bytes Transferred.
• High Intensity (>10-100 FLOPs/byte): Kernels are compute-bound (e.g., dense matrix multiplication).
• Low Intensity (<1 FLOPs/byte): Kernels are memory-bound (e.g., vector addition, element-wise operations).
• This metric determines which of the model's two fundamental limits applies.

Attainable Performance is the y-axis of the model, measured in Floating-Point Operations Per Second (FLOP/s). It represents the actual performance a kernel can achieve on a given hardware platform.

• The model plots a kernel's measured or theoretical performance point based on its operational intensity.
• The position of this point relative to the 'roofline' determines the performance ceiling and the primary limiting factor.
• The goal of optimization is to move this point as close as possible to the roofline, maximizing hardware utilization.

The Peak Compute limit is a horizontal line representing the maximum theoretical floating-point performance of the hardware, measured in FLOP/s.

• Applies to: Kernels with high operational intensity (compute-bound).
• Defined by: The processor's maximum theoretical throughput (e.g., GPU's Tensor Core FP16 FLOPs).
• Example: An NVIDIA H100 GPU has a peak FP16 Tensor Core performance of ~1979 TFLOPS. This forms the flat, upper ceiling of the roofline.
• Kernels whose performance points fall near this line are efficiently using the computational units.

The Peak Memory Bandwidth limit is a diagonal line representing the maximum rate at which data can be transferred between the processor and memory, measured in Bytes/s.

• Applies to: Kernels with low operational intensity (memory-bound).
• Defined by: The hardware's memory subsystem specs (e.g., HBM2e bandwidth).
• Slope Calculation: Attainable GFLOP/s = (Bandwidth GB/s) * (Operational Intensity FLOP/byte).
• Example: An H100 SXM has ~3.35 TB/s memory bandwidth. This forms the sloped portion of the roofline.
• Kernels on this slope are bottlenecked by data movement, not computation.

The Ridge Point (or knee) is the critical operational intensity value where the sloped memory-bound roof and the flat compute-bound roof intersect.

• Calculation: Ridge Point = Peak Compute (FLOP/s) / Peak Bandwidth (Bytes/s).
• Interpretation: It is the minimum operational intensity required for a kernel to become compute-bound. Kernels with intensity below this value are memory-bound; those above are compute-bound.
• Example: For a system with 1000 GFLOP/s peak compute and 500 GB/s peak bandwidth, the ridge point is 2 FLOPs/byte.
• This is a key hardware characteristic for determining the suitability of an architecture for a given workload.

The vertical distance between a kernel's actual performance point and the roofline represents the performance gap—the potential performance left on the table.

• Memory-Bound Optimization: To raise a point on the sloped roof, you must:
  • Increase operational intensity via kernel fusion or improved data reuse.
  • Improve effective bandwidth via memory access coalescing or better cache utilization.
• Compute-Bound Optimization: To raise a point on the flat roof, you must:
  • Increase hardware utilization via better parallelization or vectorization.
  • Use mixed-precision or specialized units (e.g., Tensor Cores).
• The model provides a clear, visual target for algorithmic and system-level tuning.

The model's central axis is operational intensity, measured in FLOPS/Byte. It is the ratio of floating-point operations performed to the volume of data moved between the processor and memory. For a given kernel:

• High Intensity: Many computations per byte loaded (e.g., large matrix multiplications in dense layers).
• Low Intensity: Few computations per byte loaded (e.g., element-wise operations, bandwidth-limited attention). Plotting a kernel's intensity determines its position on the roofline and thus its primary performance constraint.

The 'roofline' itself represents the absolute performance limit for a given hardware platform. It is defined by two key hardware characteristics:

• Peak Compute (FLOPS): The theoretical maximum floating-point operations per second of the processor (e.g., GPU's tensor cores). This forms the flat, horizontal ceiling.
• Peak Memory Bandwidth (Bytes/sec): The maximum sustainable data transfer rate between the processor and its memory (e.g., HBM2e bandwidth). This forms the diagonal, sloping ceiling. A kernel's attainable performance is the minimum of its operational intensity multiplied by memory bandwidth and the peak compute rate.

This is the primary diagnostic use case. By plotting a kernel's measured performance against its operational intensity:

• Memory-Bound (Left of Ridge Point): The kernel lies on the diagonal slope. Performance scales with intensity and is limited by memory bandwidth. Optimizations focus on improving data locality, using caches effectively, and reducing memory traffic via kernel fusion or quantization.
• Compute-Bound (Right of Ridge Point): The kernel lies on the horizontal plateau. Performance is limited by the processor's arithmetic units. Optimizations focus on improving instruction-level parallelism, using lower precision (FP16, INT8), and maximizing hardware utilization.

The vertical distance between a kernel's current performance point and the roofline ceiling represents the performance gap or available headroom. This gap quantifies the potential benefit of optimization:

• A large gap for a memory-bound kernel suggests significant gains from memory-centric optimizations.
• A kernel already near the roofline is highly optimized for that hardware; further gains require algorithmic changes to increase operational intensity or moving to new hardware with higher ceilings. It provides a data-driven answer to 'how much faster can this possibly get?'

The model is crucial for analyzing transformer inference, especially autoregressive decoding:

• Prefill Phase: High-intensity matrix multirates often make it compute-bound, benefiting from tensor core optimization.
• Decode Phase: Generating tokens one-by-one has very low operational intensity due to loading the large KV Cache for each step. This is severely memory-bound. The roofline model clearly shows why optimizing KV cache access (size, layout, bandwidth) is critical for improving Time Per Output Token (TPOT).

The model informs decisions across the stack:

• Algorithm Selection: Choosing kernels with higher inherent operational intensity for a given task.
• Precision Choice: Quantization (e.g., FP16 -> INT8) reduces the memory traffic (bytes) for the same operations, effectively increasing operational intensity and potentially moving a kernel from the memory-bound slope to the compute-bound plateau.
• Hardware Selection: Comparing rooflines for different processors (e.g., CPU vs. GPU vs. NPU) shows which is best suited for a specific model's operational intensity profile.
• Compiler Optimization: Guides kernel fusion decisions—fusing memory-bound with compute-bound ops can create a new, higher-intensity kernel.

Arithmetic Intensity is the key input variable for the Roofline Model. It is defined as the ratio of total floating-point operations (FLOPs) performed to the total bytes of data transferred from main memory. It quantifies the computational work per byte of memory traffic.

• Low Intensity (< 1 FLOP/byte): Workloads are memory-bound, limited by memory bandwidth. Examples include element-wise operations or sparse matrix computations.
• High Intensity (> 10 FLOPs/byte): Workloads are compute-bound, limited by processor FLOP/s. Examples include dense matrix multiplications or convolutions with large filter reuse.
• The Roofline Model plots attainable performance (GFLOP/s) against this metric to diagnose the primary performance limiter.

A workload or kernel is compute-bound when its performance is limited by the peak computational throughput (FLOP/s) of the processor, not by memory bandwidth. On the Roofline Model, this occurs in the region to the right of the ridge point, where the performance ceiling is a horizontal line representing peak FLOP/s.

• Characteristics: High arithmetic intensity, significant data reuse in caches, and operations like large dense matrix-matrix multiplications (GEMM).
• Optimization Focus: Efforts target improving instruction-level parallelism, using specialized tensor cores, or applying operator fusion to reduce operational overhead. The goal is to move performance closer to the theoretical 'roofline' of peak compute.

A workload or kernel is memory-bound when its performance is limited by the available memory bandwidth (bytes/sec), not by computational capacity. On the Roofline Model, this is the region to the left of the ridge point, where performance follows a sloped line determined by Bandwidth * Arithmetic Intensity.

• Characteristics: Low arithmetic intensity, frequent and irregular memory accesses with little data reuse. Common in embedding lookups, element-wise activations, and some attention operations.
• Optimization Focus: Strategies aim to improve data locality through tiling, increase cache hit rates, utilize mixed precision to reduce data movement, or apply weight pruning to create sparsity. The goal is to increase effective arithmetic intensity to become compute-bound.

In the Roofline Model, Peak Performance refers to the theoretical maximum attainable performance for a given hardware platform, represented by the two intersecting lines that form the 'roof.'

• Peak Compute (GFLOP/s): The maximum floating-point operations per second the processor can execute, defining the horizontal ceiling for compute-bound kernels. This is measured via micro-benchmarks like HPL or dense GEMM.
• Peak Memory Bandwidth (GB/s): The maximum sustainable data transfer rate between the processor and main memory (DRAM), defining the slope of the memory-bound region. This is measured with tools like STREAM benchmark.
• The ridge point is where these two limits intersect. Its x-coordinate (Peak Compute / Peak Bandwidth) is the minimum arithmetic intensity required to become compute-bound on that hardware.

Attainable Performance is the actual, measured performance (in GFLOP/s) of a specific computational kernel, plotted as a point on the Roofline graph. Its vertical distance from the roofline represents lost performance due to hardware underutilization or algorithmic inefficiencies.

• Performance Gap Analysis: If a point is far below the roof in the memory-bound region, the bottleneck is data movement. Optimization targets memory access patterns or cache usage.
• If a point is far below the roof in the compute-bound region, the bottleneck is computational efficiency. Optimization targets instruction pipelining, SIMD utilization, or kernel implementation.
• This metric is central to the model's purpose: diagnosing the nature and magnitude of performance bottlenecks to guide optimization efforts.

While often used interchangeably with Arithmetic Intensity, Operational Intensity is a more general term. The Roofline Model can be adapted to different resource constraints.

• Classic Roofline: Uses Arithmetic Intensity (FLOPs/byte) bounded by memory bandwidth.
• Extended Models: Can use other forms of operational intensity, such as:
  • Cache/Memory Hierarchy Rooflines: Plot performance against intensity defined for different cache levels (L1, L2, L3), revealing bottlenecks within the memory hierarchy.
  • Network Roofline: For distributed inference, intensity could be defined as computations per byte of network transfer, bounded by network bandwidth.
• This flexibility allows the roofline concept to diagnose bottlenecks beyond the classic CPU/GPU DRAM boundary, applicable to on-device inference with constrained cache or model serving with network overhead.