Compute-Bound

The defining metric for compute-bound workloads is arithmetic intensity, measured in FLOPs per byte of data transferred (FLOP/byte). A workload is considered compute-bound when its operational intensity exceeds the system's machine balance—the ratio of peak compute throughput to peak memory bandwidth. For example, dense matrix multiplications in large language model layers have very high arithmetic intensity, often keeping the computational units (e.g., GPU tensor cores) fully saturated while memory traffic is relatively low.

Performance scales directly with the available Floating-Point Operations Per Second (FLOPs) of the processor. Adding more memory bandwidth provides negligible benefit once the compute ceiling is hit. This is visualized using the Roofline Model, where the workload's performance plateau is under the horizontal 'compute roof' rather than the sloped 'memory roof'. Optimizations focus on maximizing FLOP utilization through techniques like:

• Operator/Kernel Fusion to reduce overhead.
• Mixed Precision Inference (e.g., FP16/BF16) to double effective FLOPs.
• Efficient use of specialized compute units like Tensor Cores or Matrix Engines.

Compute-bound behavior is most prominent during the computationally dense phases of neural network inference:

• Feed-Forward Network (FFN) Layers: The large, dense matrix multiplications are classic compute-bound operations.
• Attention Score Calculation: The Q@K^T matrix multiplication in transformer attention, especially for long sequences.
• Convolutional Layers with large kernels or high channel counts. In contrast, operations like embedding lookups or layer normalization are often memory-bound. The overall workload characteristic depends on the model architecture and the balance of these layers.

Improving performance requires increasing computational efficiency, not just reducing data movement. Key strategies include:

• Model Quantization: Reducing weight precision (e.g., to INT8) directly reduces the number of FLOPs required per operation.
• Sparsity Exploitation: Using pruned models or sparse matrix multiplication kernels to skip unnecessary calculations (zero-operand FLOPs).
• Compiler Optimizations: Advanced compilers (e.g., TensorRT, OpenXLA) perform graph-level optimizations, kernel auto-tuning, and just-in-time (JIT) compilation to generate highly efficient compute kernels for the target hardware.
• Hardware-Specific Tuning: Utilizing instructions sets (e.g., AVX-512, AMX) and data layouts (e.g., Channels-Last) optimal for the processor.

Selecting the right hardware is critical. Compute-bound workloads benefit most from processors with:

• High Peak TFLOPS (TeraFLOPs).
• Many parallel execution units (cores, streams).
• Specialized units for matrix math (e.g., NVIDIA Tensor Cores, AMD Matrix Cores, Google TPU MXUs). Performance profiling tools (e.g., Nsight Compute, Roofline model analyzers) are used to confirm the compute-bound nature by showing high ALU utilization or FPU utilization while memory bandwidth utilization remains well below peak. If memory bandwidth is saturated, the workload is memory-bound, requiring a different optimization approach.

While fundamentally limited by compute, system configuration still impacts realized performance:

• Batch Size: Increasing batch size typically improves GPU utilization for compute-bound kernels by amortizing overhead and enabling larger, more efficient matrix operations.
• Power and Thermal Limits: Sustained peak FLOP performance requires adequate cooling and power delivery. Thermal throttling can dynamically reduce clock speeds, capping achievable FLOPs.
• Kernel Launch Overhead: For very small operations, the latency to launch a GPU kernel can overshadow the compute time, making the workload latency-bound rather than purely compute-bound. Continuous batching helps mitigate this by keeping the compute units persistently busy.

The core operation in neural network inference, especially in transformer models, is large-scale matrix multiplication (matmul). In a compute-bound scenario, the arithmetic intensity—the ratio of FLOPs to memory bytes accessed—is high. This occurs when the matrices are large enough that the time spent loading weights and activations from memory is overshadowed by the time spent performing calculations.

• Transformer Feed-Forward Layers: The dense linear projections involve multiplying large weight matrices (e.g., 4096x4096) with activation vectors.
• Attention Score Calculation: The Q @ K^T operation, while often memory-bound for very long sequences, can become compute-bound when using optimized kernels (like FlashAttention) that maximize hardware utilization.
• Performance Indicator: GPU SM (Streaming Multiprocessor) utilization approaches 100%, while memory controller utilization remains lower.

While early CNN layers are often memory-bound due to large input feature maps, deeper layers with smaller spatial dimensions but many channels become compute-bound. The convolution operation involves sliding a filter across an input, performing numerous multiply-accumulate (MAC) operations per input pixel.

• Large-Kernel Convolutions: Modern architectures like ConvNeXt or models using 7x7 kernels increase the number of operations per output element, pushing workloads toward compute limits.
• Channel-Dense Layers: Layers with high channel counts (e.g., 512 or 1024 channels) have high arithmetic intensity as the number of operations grows with the product of input and output channels.
• Optimized Kernels: Using Winograd or FFT-based convolution algorithms reduces the total operation count but increases operational intensity, making the workload even more compute-saturated.

AI for science often involves models that emulate complex physical systems, which are intrinsically compute-heavy. These models perform iterative numerical computations that are not merely retrieving learned patterns but solving equations.

• Neural Operators: Models like Fourier Neural Operators (FNOs) or Graph Neural Networks (GNNs) for simulating fluid dynamics or material stress. They perform global integral transforms or message-passing aggregates across large node sets.
• Molecular Dynamics Inference: Using neural networks as force fields to predict atomic interactions. Each inference step calculates potentials and forces for thousands of atoms, involving dense network evaluations per atom or pair.
• Climate Model Emulators: Large models that map high-dimensional climate state inputs to future states require evaluating deep networks with extensive hidden layers, dominated by matrix algebra.

During the decoding phase of a large language model (LLM), after the initial prompt is processed (prefill), generating each subsequent token is often compute-bound. This is because the Key-Value (KV) Cache is already in fast GPU memory, and the primary task is a series of relatively small but dense matrix-vector multiplications.

• Small Batch Sizes: With batch size=1 (e.g., in chat applications), the operation is a matrix-vector multiply where the weight matrix is large (billions of parameters) but the vector is small. The limiting factor is the speed of the compute units, not memory bandwidth, as weights are streamed efficiently.
• High-Precision Computation: Using FP16 or BF16 precision without quantization keeps the arithmetic intensity high, as no time is spent on dequantization logic. The workload is purely about FLOPs.
• Condition: This assumes an optimized inference engine where the KV cache is managed efficiently. If the cache is constantly swapped, the workload can shift to memory-bound.

Counterintuitively, running heavily quantized models can become compute-bound. Quantization reduces memory bandwidth pressure by using lower-precision weights (INT8, FP8). However, modern AI accelerators (like NVIDIA Tensor Cores or Google TPUs) have vastly higher peak throughput for these low-precision operations.

• Saturation of Tensor Cores: When using INT8 matrix cores, the theoretical compute throughput (TOPS - Tera Operations Per Second) is so high that the workload becomes limited by how fast these units can be fed with data and execute, not by the memory subsystem.
• Dequantization Overhead: If dequantization to higher precision (e.g., INT8 -> FP16 for residual adds) is required between layers, this adds additional compute operations, increasing arithmetic intensity.
• Performance Goal: The ideal scenario where quantization achieves its goal: memory bandwidth is no longer the bottleneck, and performance scales linearly with the increased FLOP/s capacity of low-precision hardware.

In a sparsely-activated MoE model (e.g., models with 100B+ parameters), only a subset of experts (e.g., 2 out of 8) are active per token. While this reduces total FLOPs versus a dense model, the active pathway is still a dense, compute-intensive computation.

• Expert Computation: The selected experts are typically dense feed-forward networks. Processing a batch of tokens where each token routes to different experts results in a series of irregular but individually compute-bound dense matrix multiplications.
• Router Overhead: The gating/router network that selects experts is a small compute operation. The dominant cost is executing the large weight matrices of the chosen experts.
• Hardware Challenge: The compute-bound nature is complicated by the need for dynamic, conditional execution, which can lead to underutilization if not carefully scheduled, but the core kernels themselves are compute-limited.

A memory-bound inference workload is limited by the speed of data movement between the processor and memory hierarchy (e.g., GPU VRAM, CPU RAM), not by raw computational power. This occurs when the arithmetic intensity—the ratio of compute operations to memory accesses—is low.

• Key Characteristic: Performance scales with available memory bandwidth, not FLOPs.
• Common Examples: Loading model weights for small batch sizes, embedding table lookups, or operations with large, non-reusable memory footprints.
• Optimization Focus: Memory access patterns, caching strategies, and reducing data movement.

An I/O-bound (Input/Output-bound) workload is constrained by the speed of reading data from or writing data to persistent storage (disk, network) or receiving inputs over a network. The processor spends significant time waiting for data to arrive.

• Key Characteristic: Performance is gated by disk read/write speeds or network latency/bandwidth.
• Common Examples: Loading a model checkpoint from cold storage, fetching input data from a remote database, or streaming video frames over a network for inference.
• Optimization Focus: Data prefetching, using faster storage (NVMe SSDs), and network optimization.

Arithmetic Intensity is a crucial metric, defined as the number of floating-point operations (FLOPs) performed per byte of data transferred from main memory. It determines whether a kernel is compute-bound or memory-bound.

• High Arithmetic Intensity: Many FLOPs per byte. Kernels are compute-bound (e.g., large matrix multiplications in dense layers).
• Low Arithmetic Intensity: Few FLOPs per byte. Kernels are memory-bound (e.g., element-wise operations, vector embeddings).
• Roofline Model: This analytical framework uses arithmetic intensity to plot attainable performance, bounded by either peak compute (roofline) or memory bandwidth (ridge point).

The Roofline Model is an analytical performance model that visualizes the maximum attainable performance of a computational kernel or application as a function of its arithmetic intensity.

• Two Performance Limits:
  • Compute Roof: A horizontal line representing the system's peak FLOP/s. Kernels with high arithmetic intensity hit this roof (compute-bound).
  • Memory Bandwidth Ridge: A sloped line representing performance limited by memory bandwidth. Kernels with low arithmetic intensity fall under this ridge (memory-bound).
• Purpose: Provides a visual guide for optimization, showing whether to focus on improving compute efficiency or reducing memory traffic.

Latency is the time delay between submitting an inference request and receiving the complete response. For a compute-bound workload, latency is primarily determined by the time required to execute the necessary floating-point operations on the available hardware.

• Direct Impact: In a purely compute-bound scenario, reducing latency requires either more powerful hardware (higher FLOP/s) or algorithmic optimizations that reduce the total required FLOPs.
• Interaction with Bounds: While compute determines the base processing time, overall system latency can also be affected by memory access times (if memory-bound) or data loading times (if I/O-bound) that occur during the request.

Hardware Utilization measures the percentage of a processor's theoretical maximum capacity (e.g., GPU SM occupancy, CPU core usage) that is actively engaged in useful work. For a compute-bound workload, high utilization of computational units is the primary goal.

• Compute Utilization: Metrics like GPU SM Activity or FPU Usage indicate how effectively the silicon's compute resources are saturated.
• Diagnostic Tool: Low compute utilization despite high latency suggests the workload may not be purely compute-bound; bottlenecks could be elsewhere (e.g., memory stalls, poor instruction scheduling).
• Optimization Target: Techniques like kernel fusion, efficient batching, and using tensor cores aim to drive compute utilization toward 100% for compute-bound kernels.