Compute-Bound Optimization

Operational intensity measures the ratio of arithmetic operations to memory accesses (Ops/Byte). Compute-bound workloads are characterized by high operational intensity. Optimization involves restructuring computations to reuse data heavily from fast cache or registers before fetching new data from main memory. Key techniques include:

• Loop tiling/blocking: Partitioning data to fit in cache.
• Loop unrolling: Reducing loop overhead and increasing instruction-level parallelism.
• Kernel fusion: Combining multiple operations (e.g., Conv + BatchNorm + ReLU) into a single kernel to minimize intermediate memory traffic.

Modern AI accelerators contain dedicated hardware for linear algebra. Optimization requires mapping computations to these units efficiently.

• Tensor Core Mapping: Structuring matrix multiplications and convolutions to utilize dedicated, high-throughput tensor cores (e.g., on NVIDIA GPUs or Apple Neural Engine). This often requires specific data layouts (e.g., NHWC) and precision formats (FP16, BF16, INT8).
• SIMD Optimization: Using Single Instruction, Multiple Data (SIMD) instructions (e.g., NEON, AVX-512) to perform the same operation on multiple data points in parallel within a CPU core.
• Hardware Intrinsics: Writing low-level code using compiler intrinsics to directly access specific processor instructions for critical loops.

Exploiting all forms of parallelism to keep the computational units saturated.

• Data Parallelism: Distributing different data batches across multiple cores or processors.
• Model Parallelism: Splitting a single model across multiple devices (e.g., different layers on different GPUs).
• Pipeline Parallelism: Splitting the model into stages that can process different data samples concurrently.
• Instruction-Level Parallelism (ILP): Arranging independent instructions so the processor's pipeline can execute them simultaneously. Compiler auto-vectorization and careful loop structuring are critical here.

Utilizing vendor-provided tools to offload compute to dedicated accelerators.

• TensorRT: NVIDIA's SDK for optimizing models on GPUs via layer fusion, precision calibration, and kernel auto-selection.
• CoreML: Apple's framework that optimizes and compiles models for efficient execution on Apple Silicon (CPU, GPU, Neural Engine).
• NNAPI Delegation: On Android, the Neural Networks API can delegate supported model operations to NPUs, DSPs, or GPUs for accelerated, power-efficient inference.
• Vendor SDKs: Qualcomm SNPE, Intel OpenVINO, and AMD ROCm provide proprietary compilers and libraries tuned for their hardware.

Reducing numerical precision (e.g., from FP32 to INT8) directly addresses compute bottlenecks by enabling faster integer arithmetic and higher throughput on specialized units.

• Integer-Only Inference: Converts the entire graph to use integer ops, eliminating costly FPU usage and reducing power. This requires per-channel or per-tensor quantization with carefully calibrated scales.
• Mixed-Precision: Assigns higher precision (FP16) to sensitive layers and lower precision (INT8) to others, balancing accuracy and speed.
• Hardware Alignment: Choosing a quantization scheme (e.g., symmetric vs. asymmetric) that aligns with the target accelerator's native support (e.g., many NPUs prefer symmetric INT8).

The most direct application is mapping dense linear algebra operations to dedicated hardware units. This involves:

• Tensor Core Mapping: Structuring matrix multiplications (GEMM) and convolutions to utilize the systolic arrays in NPUs or the tensor cores in NVIDIA GPUs, which can deliver orders-of-magnitude higher throughput for low-precision math.
• SIMD Optimization: Using vector instructions (e.g., ARM NEON, Intel AVX-512) to process multiple data points per clock cycle for element-wise operations like activations.
• Example: A vision transformer's attention mechanism, dominated by large matrix multiplies, is restructured to run on an Apple Neural Engine or a Qualcomm Hexagon NPU, moving the workload from a general-purpose CPU.

This strategy transforms a workload to do more arithmetic per byte of data fetched from memory, pushing it deeper into the compute-bound regime. Techniques include:

• Operator Fusion: Combining sequential layers (e.g., Conv2D, BatchNorm, ReLU) into a single compiled kernel. This reduces intermediate tensor writes to slow memory, keeping data in fast registers or caches for repeated computation.
• Kernel Auto-Tuning: Automatically searching for optimal parameters like tile sizes and loop unroll factors that maximize the reuse of data loaded into on-chip memory.
• Impact: A fused convolution-activation kernel can achieve significantly higher FLOP/s utilization than executing the layers separately, as it minimizes memory bandwidth pressure.

Exploiting all available parallel execution units within a processor is fundamental. This involves:

• Data Parallelism: Splitting a large batch of inputs across multiple cores or streaming multiprocessors (SMPs).
• Model Parallelism: Partitioning a single large model's layers across different devices, used for models too large for one accelerator.
• Pipeline Parallelism: Staging different parts of the model (e.g., encoder/decoder) on different hardware units to process a stream of inputs concurrently.
• Hardware Context: On a modern mobile SoC, this means simultaneously utilizing the heterogeneous cores—CPU, GPU, and NPU—via frameworks like Android's NNAPI, which delegates subgraphs to the most appropriate unit.

AI compilers like TVM or MLIR-based frameworks apply a series of graph-level optimizations specifically for compute-bound scenarios:

• Constant Folding: Pre-computing static parts of the graph at compile time.
• Algebraic Simplification: Rewriting operations (e.g., combining reshapes) to reduce total op count.
• Layout Transformation: Converting tensor data layouts (e.g., NHWC to NCHW) to match the format expected by the most efficient hardware kernels, ensuring memory access patterns align with compute units.
• Outcome: A compiler might transform a subgraph of several operations into a single, highly optimized kernel that executes entirely on an NPU's tensor cores.

Lower numerical precision directly increases achievable FLOP/s, as hardware can perform more low-bit operations per second. This is a core synergy with model compression:

• INT8/FP16 Inference: Deploying quantized models allows the use of integer or half-precision units, which have higher theoretical peak throughput than FP32 units on most AI accelerators.
• Mixed-Precision: Assigning 16-bit precision to sensitive layers and 8-bit to others balances accuracy with the higher compute density of lower-precision math.
• Example: NVIDIA's TensorRT uses precision calibration to execute as many layers as possible in INT8 on GPU tensor cores, dramatically increasing inference speed for compute-bound models like ResNet-50.

Running a small language model (SLM) on a smartphone is a quintessential compute-bound challenge. The optimization stack involves:

• Hardware-Aware Compression: Using Quantization-Aware Training (QAT) to produce an INT8 model robust to precision loss, targeting the phone's NPU.
• Graph Compilation: Using a vendor SDK (e.g., Qualcomm SNPE or CoreML) to fuse operations, select optimal kernels, and compile the model graph for the specific SoC.
• Continuous Batching: For server-side deployment, this technique keeps the GPU's tensor cores saturated by dynamically batching incoming requests, turning a latency-sensitive task into a throughput-optimized, compute-bound workload.
• Result: The optimized model achieves minimal token generation latency by maximizing occupancy of the dedicated AI accelerator.

Low-level, hand-optimized software routines written to exploit the unique architectural features of a specific processor. These kernels are critical for achieving peak performance on AI accelerators.

• Purpose: Maximize utilization of specialized compute units like Tensor Cores (NVIDIA GPUs) or Matrix Engines (NPUs).
• Development: Often written using hardware intrinsics or assembly to control memory access patterns and instruction-level parallelism.
• Example: A convolution kernel for a mobile NPU that uses its 4x4 systolic array for 16 parallel multiply-accumulate operations per cycle.

A compiler optimization that combines multiple sequential operations into a single, fused kernel. This reduces intermediate memory traffic and kernel launch overhead, directly alleviating a common memory-bound bottleneck to reveal the underlying compute-bound limit.

• Common Fusions: Convolution + Batch Normalization + Activation (e.g., ReLU) fused into one operation.
• Benefit: Eliminates the need to write large intermediate tensors to main memory, keeping data in faster caches or registers.
• Impact: Can significantly reduce latency and improve operational intensity, making the workload more purely compute-bound.

The process of structuring computational graph operations—primarily matrix multiplications and convolutions—to efficiently utilize the dedicated, high-throughput tensor core units in modern AI accelerators.

• Requirement: Operations must be formatted with specific matrix dimensions (e.g., multiples of 16 for FP16 on NVIDIA Tensor Cores).
• Compiler Role: Frameworks like TVM or MLIR perform automatic tensor core mapping during graph compilation.
• Goal: Ensure the hardware's peak TeraFLOP/s (Trillions of Floating-Point Operations Per Second) capability is saturated.

The process of transforming a high-level neural network computational graph into a highly optimized, executable program tailored for a specific target hardware backend. This is where most compute-bound optimizations are applied.

• Key Steps: Includes operator fusion, layout transformation (e.g., NHWC to NCHW), kernel selection, and memory planning.
• Frameworks: Apache TVM, Google's XLA, and MLIR-based compilers perform this transformation.
• Output: A minimal, dependency-free executable that schedules kernels to keep the compute units constantly busy.

An automated search process that evaluates a space of possible kernel implementations and parameter configurations to find the optimal version for a given hardware platform and workload size.

• Parameters Tuned: Tile sizes, loop unrolling factors, vectorization width, and use of shared memory.
• Method: Can use brute-force search, genetic algorithms, or machine learning-based cost models.
• Benefit: Essential for portable performance, as the optimal kernel configuration varies dramatically between different CPU architectures, GPUs, and NPUs.

Compiler functions that provide direct, low-level access to specific processor instructions without writing assembly code. They are used to write the inner loops of hardware-specific kernels.

• Examples: ARM NEON intrinsics for mobile CPUs, Intel AVX-512 for servers, or proprietary intrinsics for NPU vector units.
• Function: Allow explicit control over Single Instruction, Multiple Data (SIMD) operations, enabling parallel processing of multiple data points with one instruction.
• Use Case: Manually optimizing a critical GeMM (General Matrix Multiply) routine to ensure all vector lanes are fully utilized.