Idle Cycles

The most common cause of idle cycles is processing requests with a batch size that is too small to fully saturate the GPU's parallel processing units (e.g., CUDA cores, Tensor Cores). GPUs achieve peak FLOPs (Floating Point Operations per Second) when thousands of operations can be executed in parallel. A batch size of 1 often leaves over 90% of the hardware idle. This is exacerbated by using static batching where the system waits for a fixed number of requests, leading to long queue times and underutilization during low traffic.

When sequences in a batch have different lengths, the system must pad shorter sequences to match the longest one. The GPU must compute operations for these padding tokens, which are discarded, wasting compute cycles. For example, a batch with sequences of length 10 and 100 requires 100 steps of computation for both, making 90% of the work on the first sequence useless padding. Techniques like variable-length batching or ragged tensor kernels are required to mitigate this.

During the decoding phase of autoregressive models (like LLMs), each step involves a small amount of computation on a large amount of cached data (the KV Cache). This makes the operation memory-bound; the GPU's compute units spend most of their time stalled, waiting for model weights and cache data to be loaded from High-Bandwidth Memory (HBM). The compute units are idle despite high memory controller utilization. Optimizations focus on improving cache locality and reducing data movement.

GPU execution requires synchronization points, such as between the prefilling phase and the decoding phase, or between different model layers. If work cannot be perfectly parallelized, some processor cores finish early and wait for others at these barriers. Furthermore, the inherent sequential nature of token generation (each token depends on the previous one) creates a critical path that limits parallelization, forcing most hardware to wait for that single chain of computation to complete.

Each GPU operation requires launching a kernel, which involves overhead from the driver, runtime, and memory copies between CPU and GPU. For small, frequent operations (like generating a single token), this launch overhead can be a significant fraction of the total time, during which the GPU is idle. Poorly optimized inference servers can also have substantial CPU-side bottlenecks in scheduling, batching logic, or data preprocessing, which starves the GPU of new work.

In a batch, all requests are typically locked together until the entire batch finishes. If one request has a very long sequence or experiences a slowdown, it holds up the completion and release of all other requests in that batch. This head-of-line blocking causes the GPU to remain occupied with a partially finished batch, preventing new requests from joining and utilizing the idle capacity. This is a key problem that continuous batching and iteration-level scheduling are designed to solve.

Throughput measures the total work completed per unit of time, typically in tokens per second or requests per second for inference systems. It is inversely related to idle cycles; minimizing idle time by keeping hardware constantly busy with useful work maximizes throughput. Throughput is optimized via:

• Continuous batching to maintain full batches.
• Overlapping compute with memory transfers (hiding latency).
• Efficient kernel execution to reduce stall cycles.

Tail latency (e.g., p95, p99) refers to the worst-case request completion times experienced by a small percentage of users. Idle cycles can exacerbate tail latency through head-of-line blocking, where a single long-running request in a batch stalls others. Optimization strategies include:

• Request interleaving to prioritize shorter sequences.
• Preemptive scheduling to suspend/context-switch stalled tasks.
• Predictive autoscaling to handle load spikes before queues form.

These terms classify operations by their limiting resource, which dictates where idle cycles occur.

• Memory-bound operations are limited by memory bandwidth (e.g., loading weights during decoding). The GPU compute units sit idle waiting for data.
• Compute-bound operations are limited by ALU throughput (e.g., dense matrix multiplications during prefill). The memory subsystem may be idle. Optimization involves kernel fusion to create balanced workloads and using tensor cores to accelerate compute-bound phases.

The orchestrator (cluster-level) and scheduler (instance-level) are software components responsible for eliminating idle cycles. They implement policies like continuous batching and dynamic scaling.

• The scheduler decides when to form a batch and which requests to include, aiming to keep the GPU busy.
• The orchestrator manages multiple model instances, scaling them based on queue depth to prevent idle resources during low load and overload during peaks.