Throughput

Throughput is fundamentally constrained by the compute capacity of the underlying hardware, primarily GPUs or NPUs. Key factors include:

• Compute-Bound vs. Memory-Bound Operations: Inference can be limited by raw FLOPs (compute-bound) or by the speed of reading model weights from memory (memory-bound).
• Batch Size Optimization: Larger batch sizes increase hardware utilization and throughput but also increase latency. The optimal batch size maximizes tensor core occupancy on GPUs.
• Kernel Efficiency: The performance of low-level, hardware-specific computation kernels (e.g., for matrix multiplications) provided by frameworks like CUDA or vendor SDKs.

The serving software and its batching logic are critical for aggregating work efficiently.

• Dynamic Batching: Groups multiple independent requests into a single batch for parallel execution. The maximum batch size and batch timeout are key tuning parameters.
• Continuous Batching (Iteration-Level Batching): Essential for LLMs. As used by vLLM and Triton, it adds new requests to a running batch as previous sequences finish generation, achieving near-100% GPU utilization during text generation.
• Server Overhead: The inference server's own resource consumption for request routing, scheduling, and pre/post-processing reduces available capacity for model execution.

The choice of base model and PEFT technique imposes inherent throughput characteristics.

• Model Size & Layers: Larger models (more parameters, layers) require more memory bandwidth and compute per token, reducing peak RPS.
• PEFT Inference Overhead: Methods like LoRA or adapters introduce a small but measurable computational overhead. Runtime adapter injection and multi-adapter inference allow efficient switching but add latency for weight merging or dynamic loading.
• Quantization: Deploying quantized models (e.g., FP16, INT8) via quantization-aware serving drastically reduces memory footprint and increases potential batch sizes, directly boosting throughput.

The nature of the request data and response format creates bottlenecks outside the model.

• Input Token Length: Longer prompt sequences increase the computational work for the attention mechanism and the time to first token (TTFT). Throughput in tokens/second is often a more stable metric than RPS for variable-length tasks.
• Output Token Length (for LLMs): Autoregressive generation is sequential; producing long outputs (e.g., 1000 tokens) inherently limits RPS compared to short classification tasks.
• Network Latency & Serialization: The time to transmit requests/responses over the network and serialize/deserialize data (e.g., JSON) can dominate total latency at high RPS, becoming the throughput ceiling.

The cloud or on-premise infrastructure hosting the model endpoint determines scalable capacity.

• Instance Type & Count: The number and capability (vCPUs, GPU memory) of serving instances. Autoscaling policies must react quickly to traffic spikes to maintain throughput without over-provisioning.
• Load Balancer Efficiency: A load balancer distributing requests across multiple model replicas must minimize overhead and use sticky sessions if needed for stateful adapters.
• Cold Start Latency: The delay when scaling from zero replicas, involving loading the (often large) base model into memory, which causes temporary throughput degradation.

In PEFT deployments serving multiple fine-tuned variants, throughput is shared across tasks.

• Adapter Switching Cost: In a multi-adapter inference setup, switching the active adapter per request may involve a context switch or weight reload, adding latency and reducing aggregate throughput.
• Memory Footprint per Adapter: Holding multiple adapter sets in GPU memory simultaneously enables fast switching but reduces memory available for larger batch sizes.
• Request Routing: Efficiently routing a request to a replica that already has the required adapter loaded in memory is a key optimization for maintaining high throughput in multi-tenant scenarios.

Reducing a model's numerical precision from 32-bit (FP32) to 16-bit (FP16/BF16) or 8-bit integers (INT8) via quantization slashes memory bandwidth requirements and accelerates compute, directly increasing throughput. Model compilation (e.g., with NVIDIA TensorRT or OpenAI Triton) further optimizes the quantized graph for specific hardware, fusing operations and leveraging kernel libraries to minimize latency and maximize requests per second per device.

Throughput must be maintained under variable load. Autoscaling (horizontal pod autoscaling in Kubernetes) dynamically adds or removes model inference pods based on metrics like request queue length or GPU utilization. Pair this with resource profiling to right-size CPU/memory requests per pod, preventing resource starvation or waste. Effective autoscaling ensures high throughput during peak demand while minimizing cost during low-traffic periods.

Not all requests require ultra-low latency. Decoupling request submission from result retrieval via asynchronous inference (async/await or job polling) allows the system to queue and process large, computationally expensive requests optimally. For offline or ETL pipelines, batch inference processes large datasets in bulk, maximizing throughput by leveraging full hardware capacity without the overhead of per-request API calls. Using the right pattern for the use case is key to overall system efficiency.

Latency is the time delay between submitting an inference request and receiving the prediction, typically measured in milliseconds. It is the inverse of throughput; optimizing for low latency (fast individual responses) often reduces maximum throughput. Key considerations include:

• P50, P95, P99 Latency: Percentile measurements showing the distribution of request times.
• Tail Latency: The high-percentile latencies (e.g., P99) that most impact user experience.
• Trade-off with Throughput: Batching requests improves throughput but can increase latency for the first request in the batch.

Dynamic batching is an inference optimization technique where a serving system collects multiple incoming requests into a single batch for parallel processing on hardware accelerators like GPUs. This amortizes the fixed cost of loading the model and data onto the device, dramatically increasing throughput. Implementations:

• Wait for a full batch or a timeout window before execution.
• Batch size is dynamically adjusted based on queue length.
• Essential for efficient use of GPU tensor cores, especially with large models.

Continuous batching (or iteration-level batching) is an advanced optimization for autoregressive text generation models (LLMs). Unlike static batching, it allows:

• New requests to be added to a running batch as soon as previous requests finish generating their next token.
• Finished sequences to be removed from the batch immediately, freeing memory.
• This eliminates GPU idle time, achieving near-100% utilization and maximizing throughput for variable-length generation tasks. Frameworks like vLLM and TGI implement this.

Model compilation is the process of converting a model from a framework-specific format (e.g., PyTorch) into a highly optimized, hardware-specific executable. This is a prerequisite for achieving peak throughput. Techniques include:

• Graph Optimization: Fusing operations (kernel fusion) to reduce overhead.
• Hardware Targeting: Using compilers like TensorRT, OpenVINO, or XLA to generate code for specific GPUs, CPUs, or NPUs.
• Pre-allocation: Statically planning memory and execution to minimize runtime decisions.

Autoscaling is the dynamic adjustment of compute resources (e.g., Kubernetes pods, VM instances) allocated to a model serving deployment. It maintains target throughput and latency SLAs under variable load by:

• Scaling Out: Adding more replicas of the model server to handle increased requests per second (RPS).
• Scaling In: Removing replicas during low traffic to reduce cost.
• Metrics-Driven: Triggered by CPU/GPU utilization, request queue length, or custom RPS metrics.

Cost per inference is the fundamental business metric for model serving, calculated as total infrastructure cost divided by the number of predictions served. Throughput is its primary technical driver.

• Higher Throughput dilutes fixed compute costs (e.g., GPU instance hour) across more requests, lowering cost per inference.
• Optimization Levers: Dynamic batching, model quantization, and efficient hardware use all boost throughput to reduce this cost.
• Directly ties engineering efficiency (RPS/GPU) to operational expenditure.