Roofline Model

The operational intensity of an algorithm, measured in Operations per Byte (Ops/Byte), is plotted on the logarithmic x-axis. It quantifies the balance between computation and data movement for a kernel.

• Low Intensity (< 1 Ops/Byte): Memory-bound kernels (e.g., vector addition). Performance is gated by how fast data can be loaded.
• High Intensity (> 10 Ops/Byte): Compute-bound kernels (e.g., dense matrix multiplication). Performance is gated by arithmetic units.
• This metric is algorithm-specific and independent of hardware.

The attainable performance, measured in Giga Operations Per Second (GOP/s), is plotted on the logarithmic y-axis. It represents the actual throughput achievable by a kernel on the target hardware.

• The maximum possible performance for any kernel is capped by the hardware's roofline.
• A kernel's measured performance is plotted as a single point: its operational intensity (x) and achieved GOP/s (y).
• The vertical distance from this point to the roofline shows untapped performance potential.

The memory-bound roofline is the diagonal line on the left side of the plot. Its slope equals the system's peak memory bandwidth, measured in Gigabytes per second (GB/s).

• Formula: Attainable Performance = Operational Intensity × Peak Memory Bandwidth.
• Kernels plotted in this region cannot achieve higher GOP/s without reducing data movement or increasing bandwidth.
• For TinyML, this is often the dominant constraint due to slow, power-efficient SRAM/Flash compared to compute.

The compute-bound roofline is the horizontal line at the top of the plot. Its height equals the hardware's peak computational throughput, measured in GOP/s.

• This ceiling is defined by the maximum operations per second the CPU, GPU, or NPU can execute.
• Kernels with high operational intensity hit this flat ceiling. Their performance cannot increase further without more or faster compute units.
• In practice, this peak is often for a specific precision (e.g., 8-bit integer, FP16).

Individual computational kernels (e.g., a convolutional layer) are plotted as points. Their position relative to the roofline dictates the optimization strategy.

• Point below the roofline: The kernel is inefficient. Optimization should focus on improving software: better cache use, kernel fusion, or instruction selection.
• Point on the memory-bound slope: The kernel is memory-bound. Optimize data layout, use compression, or increase data reuse.
• Point on the compute-bound ceiling: The kernel is compute-bound. Optimize with vectorization, parallelization, or using a lower precision format.

A single system can have multiple, stacked rooflines representing different hardware configurations or precision levels.

• Different Cores: One roofline for a Cortex-M4 CPU, another for an integrated micro-NPU.
• Different Numeric Precision: A lower, higher ceiling for 32-bit float ops, and a higher ceiling for 8-bit integer ops.
• Cache Hierarchies: Separate rooflines can be drawn for bandwidth from L1 cache vs. main memory, showing the performance cliff when data spills out of cache.
• This reveals the optimal hardware target and data type for a given kernel.

Operational intensity is the fundamental metric that positions an algorithm on the Roofline Model's x-axis. It is defined as the number of operations (e.g., FLOPs or MACCs) performed per byte of data transferred from main memory.

• High intensity (>10 Ops/Byte) indicates a compute-bound kernel, where performance is limited by the processor's arithmetic units.
• Low intensity (<1 Ops/Byte) indicates a memory-bound kernel, where performance is limited by memory bandwidth. For TinyML, optimizing data reuse to increase operational intensity is a key strategy to approach the compute roof.

These are the two fundamental performance regimes defined by the Roofline Model.

• A workload is compute-bound when its operational intensity is high enough that its attainable performance is capped by the hardware's peak computational throughput (the 'flat roof'). Further optimization requires improving arithmetic efficiency or using specialized compute units like NPUs.
• A workload is memory-bound when its operational intensity is low, placing it on the model's 'sloped roof.' Here, performance scales linearly with operational intensity and is limited by the available memory bandwidth. For memory-bound TinyML layers, techniques like operator fusion or improved cache locality are critical.

Peak memory usage is the maximum amount of RAM (SRAM/DRAM) consumed during inference, including model weights, activations, and intermediate buffers. It is a critical constraint in the Roofline Model's memory bandwidth calculation.

• High peak usage can force frequent data movement between slow external memory and fast caches, lowering effective bandwidth and pushing the system into a memory-bound regime.
• TinyML optimization techniques like activation sparsity, weight pruning, and efficient memory scheduling directly reduce peak usage, effectively raising the sloped memory roof for a given hardware platform.

The MACC count quantifies the computational workload of a neural network and is a primary component in calculating operational intensity for the Roofline Model.

• It represents the total number of fused multiply-and-add operations required for a forward pass.
• Combined with the total bytes of data accessed (weights + activations), it determines the algorithm's position on the x-axis: Operational Intensity = Total MACCs / Total Bytes Accessed.
• For a fixed memory bandwidth, a higher MACC count (if data access is controlled) increases operational intensity, moving the kernel closer to the compute roof.

Hardware-in-the-loop testing is the empirical methodology used to gather the precise performance data needed to construct an accurate Roofline Model for a specific TinyML deployment.

• It involves running the actual model on the target microcontroller within a controlled test environment.
• HIL testing measures the real throughput (FPS) and actual memory bandwidth utilization, which are used to plot the empirical performance point and draw the hardware's theoretical roofs.
• Without HIL validation, a Roofline analysis remains purely theoretical and may not reflect real-world bottlenecks like cache effects or DMA overhead.

Layer-wise profiling is the granular analysis technique that decomposes a model's performance according to the Roofline Model. Instead of one point for the whole network, each layer (e.g., Conv2D, Fully Connected) is analyzed individually.

• This reveals which specific layers are compute-bound and which are memory-bound, guiding targeted optimization.
• A memory-bound convolutional layer might benefit from loop tiling, while a compute-bound fully connected layer might benefit from quantization to use integer arithmetic units.
• This detailed view is essential for effective model architecture search and kernel optimization on microcontrollers.