Concurrent Requests

Concurrent request handling enables non-blocking, parallel processing where the system can work on multiple requests without waiting for any single one to complete. This is distinct from sequential execution and is fundamental to achieving high throughput.

• Key Mechanism: Utilizes asynchronous I/O, multi-threading, or multi-processing.
• Benefit: Maximizes hardware utilization (GPU/CPU) by keeping compute units busy.
• Challenge: Requires sophisticated scheduling to prevent resource starvation and ensure fairness.

An intelligent scheduler is required to decide the order and grouping of incoming requests for execution. This directly impacts latency and throughput.

• Continuous Batching: Dynamically groups requests of varying lengths into batches to maximize GPU utilization, padding only within a single batch.
• Priority Queues: Can implement different scheduling policies (e.g., First-In-First-Out, shortest-job-first) to meet specific Service Level Objectives (SLOs).
• Preemption: May allow higher-priority requests to interrupt the processing of lower-priority ones.

Concurrent requests compete for finite system resources, necessitating careful management to avoid resource contention that degrades performance.

• GPU Memory: The KV Cache for each request must be allocated and managed in shared GPU memory; inefficient management leads to fragmentation or out-of-memory errors.
• Memory Bandwidth: Multiple requests increase pressure on memory bandwidth, potentially creating a memory-bound scenario.
• Compute Cores: Scheduler must distribute work across streaming multiprocessors (SMs) to avoid under-utilization.

A core challenge is providing performance isolation between requests so that one demanding or faulty request does not adversely affect others.

• Tail Latency Amplification: Without isolation, a slow request (e.g., with a very long prompt) can block a batch, increasing P99 latency for all concurrent requests.
• Techniques: Use of separate execution contexts, memory partitions, or QoS (Quality of Service) limits.
• Measurement: Performance under concurrency is measured via percentile latencies (P90, P99), not just average latency.

For long-lived interactions like chat sessions, the system must maintain request state across multiple sequential invocations while handling many such sessions concurrently.

• Session Affinity: Ensuring subsequent requests in a conversation are routed to the same backend instance holding the session's KV Cache.
• State Management: Efficiently saving, loading, and eviction of conversation context from memory.
• Scalability Impact: Statefulness complicates horizontal scaling and load balancing compared to stateless request/response.

Systems must scale resources up or down based on the volume of concurrent requests and distribute load evenly across available hardware.

• Horizontal Scaling: Adding more model instances (replicas) to a pool to handle increased concurrency.
• Load Balancer: Distributes incoming requests across replicas; strategies include round-robin, least-connections, or latency-based.
• Autoscaling: Triggered by metrics like Queries Per Second (QPS), CPU/GPU utilization, or queue depth to maintain performance SLOs cost-effectively.

Throughput is the rate at which an inference system processes requests, measured in queries per second (QPS) or tokens per second (TPS). It represents the system's overall capacity, which is directly influenced by how well it manages concurrent requests. High throughput under concurrency is achieved through techniques like continuous batching and efficient GPU utilization.

• QPS vs. TPS: QPS measures complete requests, while TPS is specific to language models, counting generated output tokens.
• Relationship to Concurrency: Throughput typically increases with higher concurrency up to a system's saturation point, after which latency degrades.

Tail latency refers to the high-percentile latencies (e.g., P95, P99) in a request distribution, representing the slowest requests. Under high concurrency, tail latency often increases due to resource contention, variable request complexity, and scheduling inefficiencies. Managing tail latency is critical for user-perceived performance.

• Causes under Concurrency: Can be caused by memory bandwidth saturation, cache thrashing, or a single slow request blocking a batch.
• Mitigation: Techniques include request prioritization, isolation of slow paths, and over-provisioning headroom.

Continuous batching is a dynamic scheduling technique that groups multiple concurrent requests into a single computational batch, even if they start and finish at different times. This maximizes GPU utilization and throughput by eliminating idle cycles. It is a foundational optimization for serving autoregressive models like LLMs.

• Mechanism: As some requests in a batch finish generation, their slots are immediately filled with new pending requests.
• Impact on Concurrency: Enforces efficient concurrency by keeping hardware saturated, directly improving tokens per second (TPS).

Resource contention occurs when multiple concurrent requests compete for finite system resources, leading to performance degradation. This is a fundamental challenge in high-concurrency environments.

• Common Contention Points:
  • GPU Memory Bandwidth: Many requests reading/writing KV caches simultaneously.
  • CPU Cores: Handling input/output preprocessing and scheduling.
  • Host-to-Device PCIe Bus: Transferring many requests' data to the GPU.
• Effect: Increases latency and variance, particularly impacting tail latency.

The saturation point is the level of concurrent load at which a system achieves its maximum throughput. Adding more requests beyond this point does not increase throughput and causes latency to rise sharply due to queueing. Identifying this point is crucial for capacity planning and setting Service Level Objectives (SLOs).

• Identification: Found by plotting a throughput-latency curve under increasing load.
• Operational Guidance: Systems are typically run at a load slightly below saturation to maintain predictable P99 latency.

A Service Level Objective is a target for system reliability or performance. For concurrent request serving, SLOs are often defined using latency percentiles (e.g., "P99 latency < 500ms") or availability. They are the engineering goals derived from business Service Level Agreements (SLAs).

• Key SLOs for Concurrency:
  • Latency SLOs: Define acceptable P90 and P99 latency under expected concurrent load.
  • Throughput SLOs: Guarantee a minimum QPS at a defined latency.
• Measurement: Evaluated using Service Level Indicators (SLIs) like request latency.