Roofline Model

Defined as the ratio of total floating-point operations (FLOPs) performed to the total bytes of data moved between the processor and main memory (DRAM). It is measured in FLOPs/byte.

• Low Intensity: Kernels are memory-bound (e.g., element-wise operations, sparse matrix-vector multiply). Performance is limited by memory bandwidth.
• High Intensity: Kernels are compute-bound (e.g., dense matrix-matrix multiplication, convolution). Performance is limited by peak compute throughput.

The vertical axis of the plot, representing the maximum billions of floating-point operations per second (GFLOPs/sec) a kernel can achieve. This is not a single value but a curve or region determined by the kernel's operational intensity and the hardware's limits.

• For a given intensity, the attainable performance is the minimum of two values: the bandwidth-bound performance and the peak compute performance.

The critical operational intensity value where the memory-bound and compute-bound performance ceilings intersect. It is calculated as: Ridge Point = Peak Compute Performance (GFLOPs/sec) / Peak Memory Bandwidth (GB/sec)

• Kernels with intensity below the ridge point are memory-bound.
• Kernels with intensity above the ridge point are compute-bound. This point defines the hardware's inherent balance between compute and memory capabilities.

The diagonal line on the plot representing the maximum performance achievable when limited by memory bandwidth. Its slope equals the system's peak memory bandwidth.

• Formula: Attainable GFLOPs/sec = Operational Intensity (FLOPs/byte) * Peak Bandwidth (GB/sec)
• This ceiling shows that for memory-bound kernels, performance scales linearly with operational intensity. Improving performance here requires optimizing data reuse (cache locality) to increase intensity or using faster memory.

The horizontal line on the plot representing the maximum performance achievable when limited by peak computational throughput of the hardware (e.g., CPU/GPU/TPU).

• This ceiling is defined by the hardware's maximum FLOPs/sec, considering factors like vector width, core count, and clock frequency.
• Kernels in this region have high operational intensity and are limited by how fast the arithmetic units can process data already in registers or caches. Performance improvements require better utilization of SIMD units, Tensor Cores, or improving instruction-level parallelism.

Individual computational kernels are plotted as points or regions on the Roofline model based on their measured operational intensity and achieved GFLOPs/sec.

• Location Reveals Bottleneck: A point far below the roofline indicates underutilization of hardware due to suboptimal implementation, poor cache locality, or inefficient instructions.
• Optimization Vector: The vertical distance from a kernel's point to the roofline shows the headroom for optimization. The direction (moving up vs. moving right) indicates whether to focus on compute micro-optimizations or memory access patterns.

The Roofline Model is an analytical performance model that plots the attainable FLOPS (Floating-Point Operations Per Second) of a kernel against its Operational Intensity (Ops/Byte). The plot features two distinct ceilings:

• Flat, memory-bound roof: Dictated by peak memory bandwidth.
• Sloped, compute-bound roof: Dictated by peak computational throughput. A kernel's performance is represented by a single point on this plot. Its vertical distance from the roofline indicates the potential performance left on the table, guiding optimization efforts.

The model's primary application is diagnosing the fundamental limiter of an algorithm's performance on given hardware.

• Memory-Bound Kernels: Points falling on the horizontal roof. Performance is gated by how quickly data can be moved from memory. Optimization focuses on improving data reuse, cache blocking, and reducing unnecessary transfers.
• Compute-Bound Kernels: Points falling on the sloped roof. Performance is gated by the processor's arithmetic units. Optimization focuses on improving instruction-level parallelism, using vectorized instructions (e.g., AVX, NEON), and leveraging specialized units like Tensor Cores or NPUs.

The Roofline Model is essential for Hardware-Aware Neural Architecture Search (NAS) and efficient model design. Engineers use it to:

• Evaluate candidate neural network layers (e.g., standard vs. depthwise separable convolutions) for a target chip.
• Set quantitative targets for model compression techniques like pruning and quantization, aiming to shift a kernel's operational intensity rightward into the compute-bound region.
• Guide the design of custom operators and kernel fusion strategies to minimize memory traffic and maximize operational intensity.

Different hardware platforms have unique rooflines. The model forces explicit consideration of:

• Mobile CPUs/GPUs: Typically have a relatively low compute roof and moderate memory bandwidth, favoring extremely efficient operators.
• Data Center GPUs (e.g., NVIDIA H100): Feature an exceptionally high compute roof (from Tensor Cores) but require very high operational intensity to saturate it.
• Neural Processing Units (NPUs): Have rooflines shaped by their specialized matrix engines and on-chip memory hierarchies. Optimization involves tailoring dataflow and operator choices to match this profile.

For on-device inference optimization, the Roofline Model provides a first-order latency estimate: Latency ≈ (Total Operations / Attainable FLOPS). By analyzing a model's layer-by-layer operational intensity, engineers can:

• Pinpoint layers causing the most significant performance bottlenecks.
• Apply targeted optimizations like post-training quantization to reduce memory footprint and potentially increase operational intensity.
• Use the model to validate the effectiveness of compiler optimizations performed by frameworks like TVM or TensorRT.

The classic Roofline Model has limitations that practitioners must acknowledge:

• It assumes perfect overlap of compute and memory operations.
• It does not model cache hierarchies, parallelism, or power consumption.
• It treats operational intensity as a single, fixed value.

Modern extensions address these:

• Hierarchical Roofline Models: Add roofs for each level of the cache (L1, L2, LLC).
• Multi-Roofline Models: Account for different instruction types (e.g., FP32, INT8, Tensor Core ops).
• Execution-Cache-Memory (ECM) Model: A more predictive model that incorporates in-core execution time and data transfer times through the cache hierarchy.

The key independent variable in the Roofline Model. It is defined as the ratio of total floating-point operations (FLOPs) performed by a kernel to the total bytes of data moved between the processor and main memory (DRAM).

• Measured in: FLOPs/byte.
• Determines Locality: A high operational intensity indicates high data reuse (compute-bound). A low value indicates frequent memory accesses (memory-bound).
• Example: A dense matrix multiplication has high operational intensity. An element-wise vector operation has very low operational intensity.

Often used synonymously with Operational Intensity. Precisely, it is the number of arithmetic operations (not necessarily floating-point) per byte of data transfer. In the context of the Roofline Model for deep learning, FLOPs are the standard metric, making the terms functionally equivalent for performance modeling on GPUs and NPUs.

The performance limit curve in the model. It consists of two distinct regions:

• Memory-Bound Roof (Sloped): Performance is limited by the system's peak memory bandwidth. Attainable GFLOPs/sec = (Operational Intensity) × (Peak Bandwidth in GB/s).
• Compute-Bound Roof (Flat): Performance is limited by the processor's peak computational throughput. Attainable performance caps at the hardware's peak GFLOPs/sec. The intersection point where these two limits meet is the ridge point or machine balance, defining the minimum operational intensity required to achieve peak compute.

The actual performance (in GFLOPs/sec) a specific computational kernel can achieve on given hardware, as plotted within the Roofline Model. It is always at or below the roofline.

• The vertical distance from a kernel's plotted point to the roofline represents performance headroom.
• A point on the sloped roof indicates the kernel is optimally memory-bound.
• A point on the flat roof indicates the kernel is optimally compute-bound.
• A point in the interior indicates inefficiencies from suboptimal memory access patterns or underutilized compute units.

The rate at which data can be read from or written to the main memory (e.g., HBM, DDR). It is the fundamental hardware constraint for the memory-bound region of the roofline.

• Measured in: GB/s (Gigabytes per second).
• Determinants: Memory technology (DDR5, HBM3), bus width, and frequency.
• Real-World vs. Peak: The achievable bandwidth for a specific kernel depends heavily on access patterns (coalesced vs. scattered). The roofline uses the hardware's theoretical peak, but effective bandwidth can be lower.

The maximum number of floating-point operations a processor can perform per second, defining the flat ceiling of the roofline.

• Measured in: GFLOPs/sec or TFLOPS/sec.
• Determinants: Clock frequency, number of cores, and SIMD width.
• Precision Matters: Peak throughput is specified for a data type (e.g., FP32, FP16, INT8). Modern accelerators like Tensor Cores have vastly different peaks for different precisions, requiring separate rooflines for mixed-precision analysis.