Roofline Model

Operational Intensity is the primary x-axis of the roofline plot, defined as the number of operations performed per byte of data transferred from main memory. It is measured in FLOPS/Byte.

• High Intensity (>10-100 Ops/Byte): Kernels like dense matrix multiplication are compute-bound.
• Low Intensity (<1 Ops/Byte): Kernels like element-wise vector addition are memory-bound.
• This metric determines which performance ceiling (roofline) applies to a given kernel.

Attainable Performance is the y-axis of the roofline plot, representing the maximum achievable computational throughput for a kernel. It is measured in FLOPS/sec (Floating-Point Operations Per Second).

• A kernel's performance is plotted as a single point: (Operational Intensity, Attainable GFLOPs/sec).
• Its vertical position relative to the rooflines shows its efficiency. Points far below the roof indicate significant optimization headroom.

The Memory-Bound Roofline is a diagonal line representing the performance limit imposed by the system's peak memory bandwidth. Its slope equals the bandwidth (Bytes/sec).

• Formula: Attainable Performance (FLOPs/sec) = Operational Intensity (FLOPs/Byte) × Peak Memory Bandwidth (Bytes/sec)
• Kernels with low operational intensity lie on or below this line. Performance scales linearly with intensity until hitting the compute-bound roof.

The Compute-Bound Roofline is a horizontal line representing the absolute performance limit imposed by the processor's peak computational throughput.

• This ceiling is defined by the hardware's maximum FLOPS/sec, often dictated by the count and frequency of vector units or tensor cores.
• Kernels with high operational intensity are capped by this flat roof. No amount of increased intensity can yield higher performance.

The Ridge Point (or knee) is the critical operational intensity value where the memory-bound and compute-bound rooflines intersect.

• Formula: Ridge Point (FLOPs/Byte) = Peak Compute (FLOPs/sec) / Peak Bandwidth (Bytes/sec)
• This is the minimum operational intensity required to theoretically achieve peak compute performance. It is a key figure of merit for balancing system architecture.

The model can include multiple, lower Optimization Ceilings that represent performance limits for specific techniques or hardware levels.

• Example Ceilings:
  • Cache Roofline: A higher, steeper diagonal line representing the bandwidth of an on-chip cache (L1, L2).
  • SIMD Roofline: A horizontal line for performance using only scalar operations, below the peak vectorized ceiling.
• These ceilings help guide optimization efforts by showing the potential gain from improving locality or enabling vectorization.

The model's primary function is to diagnose the fundamental constraint on a kernel's performance. It plots attainable performance (GFLOPs/sec) against operational intensity (FLOPs/Byte).

• A kernel plotting below the memory-bound roof is limited by how fast data can be moved from memory (DRAM, cache).
• A kernel plotting below the compute-bound roof is limited by the processor's arithmetic units (CPU cores, NPU, GPU). For Edge AI, this instantly shows if optimization efforts should focus on improving data locality (e.g., via kernel fusion) or maximizing arithmetic throughput (e.g., via vectorization).

The Roofline Model quantitatively predicts the benefit of techniques like model quantization and weight pruning.

• Quantization (e.g., FP32 to INT8) reduces the bytes per weight/activation, effectively increasing the operational intensity of a kernel. This can move a kernel's plot to the right, potentially transitioning it from a memory-bound to a compute-bound regime, unlocking higher performance.
• The model visualizes the trade-off: while intensity increases, the compute-bound roof may lower slightly due to reduced precision throughput. Engineers use this to select the optimal quantization scheme (e.g., INT8 vs. FP16) for their specific hardware.

When selecting an edge processor (CPU, GPU, NPU, DSP), the Roofline Model provides a direct comparison of their fundamental capabilities.

• The compute-bound roof height is determined by the peak FLOPS of the accelerator.
• The memory-bound roof slope is determined by the peak memory bandwidth. By plotting the same AI kernel (e.g., a convolutional layer) on Roofline plots for different chips, a CTO can objectively determine which hardware offers the highest attainable performance for that specific workload, moving beyond vendor peak FLOPS claims to real-world potential.

On battery-powered edge devices, performance must be evaluated per watt. The Roofline Model can be extended to an Energy Roofline.

• The axes become Performance per Watt (GFLOPs/Joule) vs. Operational Intensity.
• The roofs are defined by the hardware's peak compute efficiency and memory access energy costs. This reveals the most energy-efficient operating point for a kernel. For example, running a kernel in a memory-bound regime may be fast but energy-inefficient due to costly DRAM accesses. The model guides optimizations like tiling to improve cache reuse, reducing energy-per-inference.

Compiler engineers use the model to evaluate and guide optimizations like kernel fusion, loop tiling, and vectorization.

• Kernel Fusion combines operations, reducing intermediate writes to memory. This increases operational intensity, moving the kernel rightward on the plot.
• Loop Tiling improves cache locality, effectively increasing the perceived memory bandwidth for that kernel, raising the memory-bound roof locally.
• The model answers whether a complex optimization is worthwhile: if a kernel is already compute-bound, further memory optimizations yield minimal gain, and effort should shift to improving arithmetic throughput.

Before deployment, engineers use the Roofline Model to set a theoretical performance upper bound for a model on target hardware. This becomes the baseline for SLOs (Service-Level Objectives) for latency and throughput.

• Any measured performance significantly below the Roofline indicates inefficiencies in the software stack (e.g., poor scheduling, unnecessary data copies).
• It prevents over-provisioning hardware; if a model's roofline is far below a chip's peak, a cheaper, lower-power processor may be sufficient. This data-driven approach is critical for building cost-effective, performant edge AI systems that meet deterministic execution requirements.

A workload is compute-bound when its execution time is limited by the speed of the processor's arithmetic units, not by the rate of data transfer to/from memory. On a roofline plot, this occurs at high operational intensity, where performance is capped by the horizontal peak compute throughput roof.

• Key Indicator: Performance plateaus at the compute roof.
• Optimization Target: Increase arithmetic efficiency via techniques like kernel fusion or leveraging specialized compute units (e.g., Tensor Cores).

A workload is memory-bound when its execution time is limited by the available memory bandwidth. On a roofline plot, this occurs at low operational intensity, where performance follows the sloped memory bandwidth roof.

• Key Indicator: Performance increases linearly with operational intensity along the bandwidth roof.
• Optimization Target: Reduce data movement via data reuse, improved cache locality, or model compression techniques like pruning and quantization.

Operational Intensity is the key independent variable in the roofline model, defined as the ratio of total operations (e.g., FLOPs) performed to total bytes transferred between the processor and memory (Bytes/FLOP).

• High Intensity: Workloads like dense matrix multiplication are typically compute-bound.
• Low Intensity: Workloads like element-wise operations are typically memory-bound.
• Calculation: Precisely measuring this for a kernel is critical for accurate roofline analysis.

Often used synonymously with Operational Intensity, it specifically quantifies the amount of useful arithmetic work performed per byte of data moved. It is the primary determinant of which performance roof (compute or memory) limits a given kernel.

• Bottleneck Diagnosis: Directly plots on the x-axis of the roofline model.
• Architectural Insight: Guides hardware selection; systems for low-intensity workloads prioritize high memory bandwidth, while those for high-intensity workloads prioritize peak FLOP/s.

Peak Performance is the theoretical maximum computational throughput of a hardware platform, measured in operations per second (e.g., GFLOP/s, TOPS). In the roofline model, this forms the horizontal compute roof.

• Defined by Hardware: Determined by clock frequency, number of cores, and SIMD width.
• Attainable Peak: The fraction of peak performance actually achieved by a kernel; the roofline model visualizes why this fraction is often less than 100%.

Memory Bandwidth is the maximum sustained data transfer rate between the processor and memory (e.g., GB/s). In the roofline model, this defines the slope of the memory roof.

• Critical Bottleneck: Often the limiting factor for edge AI workloads due to the high cost of off-chip data access.
• Measured via Benchmarks: Tools like STREAM or MLPerf Inference are used to establish the practical bandwidth ceiling for the roofline.