Hardware Utilization

The percentage of time the GPU's specialized matrix multiplication units (Tensor Cores) are active. These units dramatically accelerate the dense linear algebra at the heart of transformers.

• Importance: For mixed-precision inference (FP16/BF16), high Tensor Core utilization is essential for achieving peak FLOPs.
• Dependency: Requires operations to be formatted in specific sizes (e.g., 16x16 matrices) and precisions. Inefficient kernel implementations or non-conforming layer dimensions can leave this resource idle.
• Measurement: Requires deep hardware profiling (e.g., NVIDIA Nsight Compute). Low utilization here often points to suboptimal model implementation or kernel choices.

Tracks the activity of the host CPU, which manages request handling, data pre/post-processing, and orchestrates GPU tasks.

• High CPU Utilization: Can indicate the CPU is a bottleneck, struggling to feed data to the GPU fast enough, leading to GPU starvation.
• Core Imbalance: Inference servers often use dedicated CPU cores for specific tasks (e.g., a core for tokenization, a core for GPU driver calls). Poor balancing can create serialization delays.
• Context: In GPU-accelerated inference, the goal is often to minimize CPU overhead to keep the GPU fed, making CPU utilization a metric to optimize downward in many cases.

The instantaneous electrical power consumed by the accelerator (GPU) or entire system. It is a direct driver of operational cost and cooling requirements.

• Efficiency Metric: Performance per Watt (e.g., Tokens/Second/Watt) is a key metric for total cost of ownership (TCO).
• Dynamic Range: GPUs draw less power at idle and scale up under load. Sustained operation near the Thermal Design Power (TDP) limit indicates full utilization but requires robust cooling.
• Use Case: Critical for edge AI and data center planning. Techniques like model quantization directly reduce computational intensity, lowering power draw.

The data transfer rate across the interconnect between the host CPU and GPU (PCIe) or between multiple GPUs (NVLink).

• Bottleneck Identification: Low utilization of this link during inference suggests the model fits entirely within GPU memory, and data transfer is minimal. High utilization may indicate frequent CPU-GPU data swaps, which is detrimental to latency.
• Multi-GPU Inference: For model-parallel or pipeline-parallel execution, NVLink utilization is critical for minimizing the latency of transferring activations and gradients between GPUs.
• Impact: Directly affects Time to First Token (TTFT) if the initial prompt or model weights must be transferred to the GPU.

A compute-bound inference workload is limited by the available computational capacity (FLOPs) of the processor rather than by memory bandwidth or I/O operations. This occurs when the model's arithmetic intensity is high, meaning it performs many calculations per byte of data fetched from memory.

• Characteristics: High GPU/CPU core utilization, low memory traffic.
• Optimization Focus: Improving kernel efficiency, using tensor cores, and operator fusion.
• Example: Dense matrix multiplications in large transformer layers.

A memory-bound inference workload is limited by the speed at which data can be transferred between processor cores and memory (bandwidth) rather than by raw computational power. This is typical for operations with low arithmetic intensity.

• Characteristics: High memory bandwidth utilization, processor cores often waiting for data.
• Optimization Focus: Improving data locality, cache utilization, and weight quantization to reduce memory footprint.
• Example: Loading model weights or performing element-wise operations on large tensors.

The roofline model is an analytical performance model that visualizes the attainable performance of a computational kernel. It plots performance (FLOPs/sec) against operational intensity (FLOPs/byte).

• Key Components: A ceiling defined by peak compute throughput and a slope defined by peak memory bandwidth.
• Purpose: Identifies whether a kernel is compute-bound or memory-bound and quantifies its distance from optimal hardware utilization.
• Use Case: Guides optimization efforts by showing if you should focus on improving compute efficiency or reducing memory traffic.

Resource contention occurs when multiple inference requests, processes, or threads compete for access to a finite, shared system resource, leading to performance degradation for all parties.

• Common Contention Points: GPU memory bandwidth, shared L2/L3 cache, PCIe bus bandwidth, and CPU cores.
• Impact: Increased and variable latency, reduced overall throughput.
• Mitigation: Techniques like careful GPU memory allocation, efficient scheduling (e.g., NVIDIA MPS), and isolating high-priority workloads.

GPU Memory Optimization encompasses techniques for efficient allocation, management, and usage of high-bandwidth memory on accelerator cards during inference.

• Key Techniques:
  • PagedAttention/KV Cache Management: Efficiently stores attention keys and values.
  • Memory Pooling: Reuses memory blocks to avoid allocation overhead.
  • Unified Memory: Allows oversubscription of GPU memory with system RAM, albeit with a performance penalty.
• Goal: Maximize the effective batch size and sequence length that can fit on a device to improve throughput and utilization.

Operator/Kernel Fusion is a compiler-level optimization that combines multiple, sequential low-level computational operations into a single, fused kernel executed on the GPU or CPU.

• Benefit: Dramatically reduces memory-bound bottlenecks by keeping intermediate results in fast registers or caches instead of writing them to slow main memory.
• Example: Fusing a GeLU activation function with the preceding matrix multiplication.
• Frameworks: Enabled by compilers like XLA (TensorFlow/JAX), TorchInductor (PyTorch), and Triton.