Compute-Bound

A compute-bound operation is characterized by high arithmetic intensity, defined as the ratio of arithmetic operations (FLOPs) to bytes of data transferred from memory. In dense matrix multiplications (matmuls), common in the prefill phase of transformer inference, each weight value is reused many times for calculations, making the operation compute-saturated. The performance is gated by the speed of the Arithmetic Logic Units (ALUs) in the GPU's streaming multiprocessors, not by the memory bandwidth.

The prefill or context encoding phase of autoregressive models like LLMs is a classic compute-bound workload. The entire input prompt is processed in parallel through the model's layers, performing massive, contiguous matrix multiplications between the input tokens and the full weight matrices. This phase can consume over 90% of the total FLOPs for a request but only occurs once per sequence, making its efficient scheduling via techniques like continuous batching critical for overall system throughput.

When an operation is compute-bound, the GPU's streaming multiprocessors (SMs) are kept busy with continuous calculations, leading to high ALU utilization. Monitoring tools will show:

• High SM Activity (e.g., >80%).
• Lower Memory Controller utilization.
• Performance that scales nearly linearly with increased clock speed or core count. Optimization focuses on ensuring the GPU's compute resources are fully saturated, often by increasing the batch size to provide more parallel work, using tensor cores for mixed-precision math, and employing kernel fusion to reduce overhead.

It is defined in contrast to memory-bound operations. The key differentiator is the limiting factor:

• Compute-Bound: Limited by FLOPs/sec. Performance improves with faster math units.
• Memory-Bound: Limited by bytes/sec. Performance improves with higher memory bandwidth. In transformer inference, the decoding phase is typically memory-bound due to the small, sequential reads of the KV Cache. An optimal inference engine must manage both types of workloads: using large, compute-bound batches for prefill, and efficient, memory-bound scheduling for decoding.

To maximize performance for compute-bound workloads, engineers employ:

• Batching: Aggregating multiple requests to create larger, more efficient matrix operations.
• Kernel Fusion: Combining multiple sequential operations (e.g., activation + bias) into a single GPU kernel to reduce launch overhead and improve instruction cache locality.
• Mixed Precision: Using lower-precision formats like FP16 or BF16 with Tensor Cores, which can double or quadruple theoretical FLOPs/sec.
• Operator Optimization: Using hardware-specific, hand-tuned libraries (e.g., cuBLAS, CUTLASS) for fundamental operations like GEMM.

Compute-bound workloads directly benefit from hardware advancements that increase raw computational throughput:

• More / Faster Cores: Higher FLOPs capacity.
• Specialized Units: Tensor Cores (NVIDIA) or Matrix Cores (AMD) that accelerate mixed-precision matrix math.
• Increased Clock Speeds: Directly reduces computation time. They benefit less from improvements in memory bandwidth or cache size once a minimum threshold is met to feed the compute units. This makes chip architecture analysis crucial for inference cost forecasting.

This technique combines multiple, sequential low-level computational operations (kernels) into a single, larger kernel. The primary benefit is the elimination of kernel launch overhead and the reduction of intermediate data writes to global memory. For compute-bound operations like a series of element-wise activations following a matrix multiplication, fusion keeps data in fast on-chip registers or shared memory, dramatically increasing arithmetic intensity (FLOPs per byte of memory access). Frameworks like TensorRT and XLA perform automatic kernel fusion during graph compilation.

Leveraging lower numerical precision formats (e.g., FP16 or BF16) for compute-intensive operations. Modern GPU tensor cores (e.g., NVIDIA's Tensor Cores, AMD's Matrix Cores) perform these lower-precision matrix multiplications with significantly higher throughput (in TFLOPs) compared to full FP32 precision. This directly accelerates the core compute-bound math. Critical techniques include:

• Automatic Mixed Precision (AMP): Using FP16 for forward/backward passes while maintaining a master FP32 copy of weights for stability.
• Quantization-Aware Training (QAT): Training the model with simulated lower precision (e.g., INT8) to maintain accuracy before deployment.
• Post-Training Quantization (PTQ): Converting a trained FP32 model to INT8 using calibration data.

Explicitly structuring computations to maximize the use of dedicated matrix multiplication hardware. This involves ensuring matrix dimensions align with the hardware's preferred tile sizes (e.g., multiples of 16 or 32 for NVIDIA Tensor Cores). Performance cliffs occur with suboptimal shapes. Optimization strategies include:

• Padding and Alignment: Adjusting input dimensions minimally to meet hardware requirements.
• Kernel Selection: Using specialized cuBLAS or CUTLASS kernels designed for specific shape and precision combinations.
• Block Sparse Kernels: For pruned models, using kernels that skip zero blocks, performing dense computation only on active regions, thus making effective use of compute units.

Ahead-of-time compilation of a neural network's computational graph into an optimized, static execution plan. This allows the compiler (e.g., TVM, XLA, TensorRT) to perform global optimizations that are impossible with eager execution:

• Aggressive Constant Folding: Pre-computing static portions of the graph.
• Common Subexpression Elimination: Reusing the results of identical computations.
• Optimal Kernel Scheduling: Minimizing idle time between kernels by analyzing dependencies and maximizing instruction-level parallelism (ILP).
• Memory Planning: Pre-allocating all intermediate tensors in a fixed, reusable memory pool to eliminate dynamic allocation overhead during inference.

Exploiting multiple dimensions of parallelism inherent in compute-bound tensor operations to saturate all available compute resources.

• Data Parallelism: Distributing different samples in a batch across multiple GPU SMs. This is the primary parallelism for batch inference.
• Model Parallelism: Splitting the layers of a single model across multiple devices (e.g., pipeline parallelism).
• Tensor Parallelism: Splitting individual weight matrices and the associated computation across devices, requiring communication per layer.
• Operation Parallelism: Execuring independent sub-graphs of the model concurrently on the same device where possible.

Replacing exact, compute-heavy mathematical operations with faster approximations that maintain sufficient accuracy for the task. This reduces the fundamental FLOP count of the algorithm.

• Approximate Activation Functions: Using fast, piecewise-linear approximations for functions like GELU or SiLU.
• Low-Rank Decompositions: Factorizing large weight matrices (e.g., a d x d feed-forward layer) into the product of two smaller matrices (d x r and r x d), reducing computation from O(d²) to O(2dr).
• Structured Pruning: Removing entire channels, neurons, or attention heads to create a smaller, denser, and more compute-efficient model architecture.

A memory-bound operation is one whose execution time is limited by the speed of reading from or writing to memory (e.g., DRAM, HBM), rather than by the processor's computational throughput. This is the typical characteristic of the autoregressive decoding phase in transformer inference, where generating each new token involves loading the large KV cache and model weights, with relatively little arithmetic. The performance is gated by memory bandwidth, not FLOPs.

• Primary Example: Token-by-token generation in LLMs.
• Contrast: The opposite of a compute-bound operation.
• Optimization Focus: Memory hierarchy, cache locality, and bandwidth.

Arithmetic Intensity is a key hardware/software metric defined as the number of arithmetic operations performed per byte of data moved from main memory. It determines whether an operation is compute-bound or memory-bound.

• High Arithmetic Intensity: Operations like dense matrix multiplication are compute-bound, as they perform many FLOPs on each loaded data element.
• Low Arithmetic Intensity: Element-wise operations are memory-bound, as they perform few FLOPs per loaded byte.
• Roofline Model: This analytical model uses arithmetic intensity to plot attainable performance against hardware limits (compute peak and memory bandwidth).

The prefill phase (or context encoding phase) is the initial, parallel processing stage in autoregressive model inference where the entire input prompt is processed at once. This phase is typically compute-bound.

• Characteristics: Involves large, dense matrix multiplications across the full sequence length.
• Contrast with Decoding: Followed by the memory-bound decoding phase.
• Optimization Impact: Techniques like continuous batching are crucial here to maintain high GPU utilization by aggregating compute from multiple requests.

Kernel Fusion (or operator fusion) is a compiler-level optimization that combines multiple, sequential low-level computational operations (kernels) into a single, compound kernel. This is a critical technique for alleviating memory-bound bottlenecks and can help create more compute-bound workloads.

• Mechanism: Reduces intermediate writes to and reads from GPU global memory.
• Benefit: Decreases kernel launch overhead and increases arithmetic intensity by keeping data in fast registers or shared memory.
• Example: Fusing a GeLU activation function into the preceding matrix multiplication kernel.

Model Quantization is a model compression technique that reduces the numerical precision of a model's weights and activations (e.g., from 32-bit floating-point to 8-bit integers). This directly addresses both compute and memory bottlenecks.

• Compute-Bound Impact: Lower precision enables more operations per second (higher theoretical FLOPs) and can reduce latency for compute-bound layers.
• Memory-Bound Impact: Drastically reduces the model's memory footprint, decreasing bandwidth pressure during the decoding phase.
• Types: Includes post-training quantization (PTQ) and quantization-aware training (QAT).

The Roofline Model is an analytical performance model used to visualize the attainable performance of a computational kernel or algorithm on a given hardware platform. It explicitly distinguishes between compute-bound and memory-bound regimes.

• Axes: Plots performance (FLOPs/sec) against arithmetic intensity (FLOPs/byte).
• The Roofline: The attainable performance is capped by either the hardware's peak compute capacity (a horizontal line) or its memory bandwidth multiplied by arithmetic intensity (a sloping line).
• Use Case: Guides optimization efforts by identifying the limiting factor for a specific operation.