Bottleneck Analysis

Bottleneck analysis is not guesswork; it is a structured, data-driven process. It involves instrumenting the entire inference pipeline—from request ingress and pre-processing, through model execution on hardware, to post-processing and response egress—to collect granular timing and resource utilization metrics. The goal is to move from observing a symptom (e.g., "high latency") to identifying the precise root cause (e.g., "GPU memory bandwidth saturation"). This requires correlating metrics across software layers and hardware components to isolate the limiting factor.

The core output of the analysis is pinpointing the specific resource or component that forms the bottleneck. Common categories include:

• Compute-Bound: Limited by the processor's FLOPs (e.g., dense matrix multiplications).
• Memory-Bound: Limited by memory bandwidth or capacity (e.g., loading model weights, KV cache thrashing).
• I/O-Bound: Limited by disk or network speed (e.g., loading model checkpoints, fetching context from a vector database).
• Synchronization-Bound: Limited by inter-process communication or locking (e.g., in multi-GPU or distributed inference). Tools like the Roofline Model help classify workloads by plotting operational intensity against attainable performance, clearly showing whether the bottleneck is compute or memory.

Effective analysis relies on performance profilers that provide low-overhead, detailed measurements. Key tools and metrics include:

• GPU Profilers (e.g., NVIDIA Nsight Systems, PyTorch Profiler): Measure kernel execution times, memory copies, and SM (Streaming Multiprocessor) utilization.
• CPU Profilers (e.g., perf, VTune): Identify hot functions and cache misses in pre/post-processing code.
• System Monitoring (e.g., Prometheus, Grafana): Track high-level metrics like GPU utilization %, memory usage, and network I/O.
• Application-Level Tracing: Instrument code to measure latency of specific pipeline stages (e.g., tokenization, inference, detokenization). The analysis correlates data from all these sources.

A bottleneck is not an intrinsic property of a system but is revealed under specific load conditions. Analysis must be performed across the throughput-latency curve, from low load to the saturation point. A system may be compute-bound under low concurrency but become memory-bound due to resource contention when handling many concurrent requests. Understanding how the bottleneck shifts with load—such as the transition from being limited by Time to First Token (TTFT) to being limited by Time per Output Token (TPOT)—is critical for capacity planning and optimization.

The ultimate purpose of bottleneck analysis is to prescribe targeted optimizations. The identified bottleneck directly dictates the remediation strategy:

• If Compute-Bound: Explore model quantization, kernel fusion, or using more powerful hardware.
• If Memory-Bound: Implement KV cache management, continuous batching, or weight pruning.
• If I/O-Bound: Optimize model loading with faster storage or implement smarter caching.
• If Latency is in Tail (P99): Investigate resource contention or garbage collection pauses. Without this analysis, optimization efforts are speculative and often ineffective.

Bottleneck analysis is iterative. After applying an optimization, the analysis must be re-run to:

1. Verify the original bottleneck has been alleviated.
2. Identify the new bottleneck that now limits performance (Amdahl's Law). It is also comparative. Establishing a performance baseline under a synthetic workload allows for clear measurement of performance regression or improvement. Comparing profiles before and after a change (e.g., a new model version or framework update) is essential for maintaining Service Level Objectives (SLOs).

A software tool that instruments code to collect detailed timing and resource usage data during execution. Profilers are essential for bottleneck analysis as they identify hotspots—specific functions, operators, or kernel calls consuming disproportionate time or resources. Common outputs include:

• Flame graphs visualizing call stack frequency.
• GPU trace timelines showing kernel execution and memory transfers.
• CPU and memory sampling to pinpoint inefficient code paths. Tools like NVIDIA Nsight Systems, PyTorch Profiler, and TensorFlow Profiler provide this granular visibility.

An analytical performance model that visualizes the maximum attainable performance of a computational kernel or model layer. It plots performance (e.g., FLOPs/sec) against operational intensity (FLOPs/byte). The model reveals whether a kernel is compute-bound (limited by peak FLOPs) or memory-bound (limited by memory bandwidth).

• If a kernel's performance point falls below the compute roof, it's compute-bound; optimization focuses on improving arithmetic efficiency.
• If below the memory roof, it's memory-bound; optimization focuses on data reuse, fusion, and memory access patterns. This model provides a theoretical ceiling for performance, guiding bottleneck analysis toward the correct class of optimization.

A set of metrics measuring the percentage of available hardware resources actively engaged during inference. Low utilization often indicates a bottleneck elsewhere in the system. Key metrics include:

• GPU Utilization: Percentage of time one or more GPU kernels are executing. Sustained low utilization may indicate a CPU or I/O bottleneck.
• GPU Memory Utilization: Percentage of allocated GPU memory. High usage can lead to expensive swapping.
• CPU Utilization: Can be broken into user and system time. High system time may indicate excessive kernel launches or I/O wait.
• Memory Bandwidth Utilization: Percentage of peak theoretical bandwidth used, critical for identifying memory-bound workloads. Tools like nvidia-smi and vmstat provide these measurements.

A fundamental graph that plots the relationship between system throughput (e.g., Queries Per Second) and request latency (e.g., P99) as the offered load increases. This curve is the primary output of load testing and directly reveals system bottlenecks.

• Linear Region: Latency remains stable as throughput increases; the system is underutilized.
• Knee of the Curve: Latency begins to increase noticeably; the system is approaching its saturation point.
• Saturation Region: Throughput plateaus at its maximum while latency increases sharply; the primary bottleneck is fully saturated (e.g., GPU compute, memory bandwidth). Analyzing how this curve changes after an optimization confirms whether the targeted bottleneck was correctly identified and alleviated.

The degradation in performance that occurs when multiple inference requests, processes, or system tasks compete for access to a finite shared resource. It is a common source of bottlenecks in multi-tenant serving environments. Key types include:

• Memory Contention: Multiple processes/page faults causing slow DRAM or GPU memory access.
• Compute Contention: Multiple kernels or processes competing for GPU SMs or CPU cores, leading to context-switching overhead.
• I/O Contention: Multiple models or data loaders competing for disk or network bandwidth.
• Cache Contention: Different workloads evicting each other's data from shared CPU/GPU caches. Bottleneck analysis must distinguish between a resource being inherently slow and it being slow due to contention from other work.