Compute Bound vs. Memory Bound

When performance is limited by the arithmetic logic units (ALUs), the goal is to maximize the number of operations per clock cycle. Key strategies include:

• Operator Fusion: Combining consecutive layers (e.g., Conv2D + BatchNorm + ReLU) into a single kernel to eliminate intermediate memory writes and keep data in registers.
• Loop Unrolling & Tiling: Restructuring computation loops to increase instruction-level parallelism and improve pipeline utilization.
• Leveraging SIMD/Vector Units: Using Single Instruction, Multiple Data (SIMD) instructions to perform multiple multiply-accumulate (MAC) operations in a single cycle.
• Quantization to Lower Bitwidth: Moving from 32-bit floating-point to 8-bit integer (INT8) or binary operations drastically increases the theoretical operations per second (OPS) the hardware can perform.
• Using Hardware Accelerators: Offloading dense matrix multiplications (GEMM) or convolutions to a dedicated Neural Processing Unit (NPU) or Digital Signal Processor (DSP) block.

When performance is limited by data movement bandwidth, the goal is to minimize and localize memory accesses. Key strategies include:

• Weight Pruning & Sparsity: Removing redundant weights (setting them to zero) reduces the model size that must be loaded from flash into RAM, decreasing the memory footprint and access latency.
• Activation Compression: Applying techniques like activation quantization or using compact data formats (e.g., INT8 activations) to reduce the size of feature maps transferred between layers.
• Memory Layout Optimization (Data Locality): Organizing tensor data in memory-contiguous, cache-friendly formats (e.g., NHWC vs. NCHW) to maximize cache hit rates and minimize stalls.
• Layer/Operator Fusion: A critical technique for memory-bound systems; fusing layers avoids writing large intermediate activation tensors to slow main memory (SRAM/DRAM) and instead keeps them in faster registers or cache.
• Smart Caching & Prefetching: Designing the inference scheduler to proactively load the weights for the next layer while the current layer is computing, hiding memory latency.

The Roofline Model is an essential analytical framework for identifying the bound and guiding optimization. It plots attainable performance (Giga-OPS/sec) against operational intensity (OPS/byte).

• The Compute Roof: A horizontal line representing the hardware's peak computational throughput.
• The Memory Roof: A diagonal line whose slope is the system's peak memory bandwidth.
• Plotting a Kernel: A model layer's operational intensity and measured performance are plotted as a point.
  • If the point is near and limited by the horizontal compute roof, the kernel is compute-bound.
  • If the point is near and limited by the diagonal memory roof, the kernel is memory-bound.
• Optimization Vector: The model shows the potential gain. For a memory-bound point, moving vertically toward the compute roof requires increasing operational intensity (e.g., via fusion).

The optimal strategy depends heavily on the target microcontroller's architecture.

• For CPUs with Large Caches (e.g., Cortex-M7): Focus on data locality and cache blocking to keep working sets within L1 cache. Loop tiling is highly effective.
• For CPUs with Tight SRAM (e.g., Cortex-M4): Memory footprint is paramount. Aggressive pruning, 8-bit quantization, and layer fusion are necessary to fit the model and activations in limited SRAM.
• For Systems with Flash-based Weight Fetch: If weights are stored in slow flash memory, performance is often memory-bound by flash bandwidth. Use asynchronous DMA transfers to fetch weights for the next layer during computation.
• For NPU/Accelerator-based Systems (e.g., Ethos-U55): The system may become memory-bound feeding the accelerator. Optimize by ensuring a continuous stream of data via efficient DMA setups and minimizing CPU-Accelerator synchronization overhead.

Accurate diagnosis requires measurement, not guesswork. Use profiling tools to collect key metrics:

• Cycle Counts & IPC: Low Instructions Per Cycle (IPC) often indicates memory stalls, suggesting a memory bound.
• Cache Miss Rates: High L1/L2 cache miss rates from performance counters are a direct signal of memory-bound behavior.
• Memory Bus Utilization: Tools that monitor the AHB/AXI bus can show if memory bandwidth is saturated.
• Layer-wise Profiling: Break down the inference time and memory traffic per layer. Often, 80% of the latency comes from 20% of the layers (e.g., large fully-connected or early convolutional layers), which become the primary optimization targets.
• Tools: ARM Streamline, Segger SystemView, or custom CMSIS-DSP/Perf counters provide this low-level data.

Many optimization techniques for compute or memory bounds involve a trade-off with model accuracy. The engineer must navigate the Pareto frontier.

• For Compute-Bound Systems: Heavy quantization (e.g., to INT4) or aggressive pruning increases OPS/sec but risks significant accuracy loss, requiring careful quantization-aware training or fine-tuning.
• For Memory-Bound Systems: Pruning and activation compression reduce memory traffic but can degrade accuracy. Structured pruning often preserves accuracy better than unstructured pruning.
• Holistic Co-design: The most effective approach is hardware-aware neural architecture search (HW-NAS), which searches for model architectures that are inherently efficient (high operational intensity, low memory footprint) for the target hardware constraint, finding optimal points on the accuracy-latency-memory Pareto frontier.

Operational intensity is a key metric that quantifies the balance between computation and memory access in an algorithm. It is defined as the number of operations (e.g., FLOPs or MACCs) performed per byte of data transferred from main memory.

• A high operational intensity indicates a compute-bound kernel, where the processor can keep its arithmetic units busy with data already in fast caches.
• A low operational intensity indicates a memory-bound kernel, where performance is gated by waiting for data to be fetched from slow, off-chip memory.

This metric is central to the Roofline Model, which uses it to predict the maximum attainable performance of a computational kernel on specific hardware.

The Roofline Model is an analytical performance model that visualizes the attainable performance of a computational kernel or a full neural network. It plots performance (e.g., Giga Operations Per Second) against operational intensity.

The model establishes two fundamental "roofs":

• A memory-bound roof, defined by the hardware's peak memory bandwidth.
• A compute-bound roof, defined by the hardware's peak computational throughput.

A kernel's performance is limited by whichever roof it hits first. For TinyML, this model is crucial for diagnosing whether a layer's poor performance is due to insufficient compute speed or constrained memory bandwidth, guiding optimization efforts like kernel fusion or quantization.

Layer-wise profiling is the detailed measurement and analysis of resource consumption for each individual layer or operator within a neural network. This granular data is essential for diagnosing compute-bound vs. memory-bound bottlenecks.

Key metrics profiled per layer include:

• Execution Time: Identifies latency-heavy layers.
• Peak Memory Usage: Tracks activation and weight memory, highlighting layers that cause large data transfers.
• MACC/FLOP Count: Quantifies computational workload.
• Data Movement Volume: Measures bytes read/written from DRAM.

By analyzing this profile, developers can pinpoint if a convolutional layer is slow due to a high MACC count (compute-bound) or because its large feature maps exceed cache capacity, forcing frequent DRAM accesses (memory-bound).

Memory bandwidth is the maximum rate at which data can be read from or written to a memory subsystem by a processor, typically measured in gigabytes per second (GB/s). It is the critical hardware limit for memory-bound systems.

In microcontroller systems, the hierarchy is stark:

• SRAM/TCAM (On-Chip): Fast, low-latency, but limited capacity (10s-100s of KB). Bandwidth can be high.
• Flash (On-Chip): Very slow for reads, used primarily for storing model weights.
• External DRAM: Higher capacity but with significant latency and power penalty for access.

A memory-bound workload saturates this available bandwidth. Optimization techniques focus on reducing the working set size (via quantization, pruning) and improving data locality to stay in faster on-chip memory.

Peak compute throughput is the theoretical maximum number of operations a processor can perform per second, often measured in Giga Operations Per Second (GOPS) or Giga Multiply-Accumulates per Second (GMACS). It is the critical hardware limit for compute-bound systems.

This limit is determined by:

• Clock Frequency: The processor's cycle speed.
• Parallelism: The number of arithmetic logic units (ALUs), vector units (SIMD), or dedicated neural processing unit (NPU) cores.
• Data Width: The bit-width of operations (e.g., 8-bit INT8 vs. 32-bit FP32).

A compute-bound workload saturates these arithmetic units. Optimization involves leveraging hardware intrinsics, efficient kernel libraries, and model architectures that maximize the use of available parallel compute resources.

The cache miss rate is the percentage of memory accesses where the requested data is not found in a fast cache and must be fetched from a slower, higher-level cache or main memory. A high cache miss rate is a primary indicator of a memory-bound condition.

For TinyML on MCUs with small caches (often just a few KB):

• Weights: If the active weight tile doesn't fit in cache, each miss stalls the compute pipeline.
• Activations: Large intermediate feature maps can evict other needed data, causing thrashing.

Profiling cache misses (via performance counters or simulation) reveals whether performance is limited by computation or by waiting for data. Techniques like loop tiling, layer fusion, and activation compression are designed specifically to reduce the cache miss rate.