Orchestrator

The orchestrator's primary function is to schedule incoming requests onto available model workers. It implements batching policies—such as dynamic batching or continuous batching—to group multiple queries for parallel execution, maximizing GPU utilization. It decides when to form a batch based on configurable parameters like a batch window or queue depth, balancing throughput gains against added latency for individual requests.

The orchestrator manages the entire lifecycle of model instances on inference servers. This includes:

• Scaling: Automatically scaling the number of active model replicas up or down based on request load.
• Health Checking: Continuously monitoring worker health and restarting failed instances.
• Version Management: Seamlessly rolling out new model versions (e.g., A/B testing, canary deployments) and draining traffic from old versions.
• Resource Allocation: Assigning models to specific hardware (e.g., GPUs, NPUs) and managing memory allocation.

The orchestrator distributes incoming inference requests across a pool of identical model workers to prevent any single instance from becoming a bottleneck. It employs strategies like:

• Round-robin or least-connections routing.
• Intelligent routing based on real-time metrics like queue depth or latency.
• Request interleaving to mix short and long requests efficiently. It also implements request admission control and load shedding to reject traffic when the system is overloaded, protecting stability.

The orchestrator enforces Quality of Service (QoS) by optimizing for key performance indicators. Its responsibilities include:

• Latency Management: Implementing policies like batch timeouts to bound tail latency (p95, p99).
• Throughput Maximization: Continuously tuning batching parameters to achieve the highest possible tokens/sec.
• Cost Control: Optimizing for metrics like cost-per-token by maximizing hardware efficiency and minimizing idle cycles.
• Priority Queuing: Ensuring high-priority requests are serviced ahead of background batch jobs.

For stateful inference, particularly with autoregressive models like LLMs, the orchestrator is responsible for managing the key-value (KV) cache. This involves:

• Efficiently allocating and freeing GPU memory for cache per request.
• Implementing cache eviction policies when memory is constrained.
• Orchestrating the prefilling phase (parallel prompt processing) and the decoding phase (sequential token generation) across requests.
• Managing attention sinks or other cache optimization techniques to handle long contexts.

The orchestrator provides the central point for monitoring the inference cluster. It collects and exposes critical telemetry, including:

• System Metrics: GPU utilization, memory usage, and queue lengths.
• Business Metrics: Request per second (RPS), tokens per second, and error rates.
• Performance Metrics: Per-request latency, iteration time, and batch size distributions.
• Cost Metrics: Inference cost per request, often broken down by model and user. This data is essential for inference performance benchmarking and cost optimization.

A batching policy is the set of deterministic rules and heuristics used by an orchestrator's scheduler to decide when and how to group requests. It is the algorithmic core of request optimization. Policies balance competing objectives:

• Throughput vs. Tail Latency
• GPU Utilization vs. Request Fairness
• Static Batching: Fixed batch formed, executes, and returns all results together.
• Dynamic Batching: Batches formed based on a time window or queue size.
• Continuous Batching: The most advanced policy, allowing iteration-level scheduling where requests dynamically join and exit the batch.

Request admission control is a policy mechanism where the orchestrator decides to accept, queue, or reject incoming inference queries based on real-time system state. This is a critical function for maintaining service-level agreements (SLAs) and preventing system overload. It evaluates:

• Current system load and available capacity (GPU memory, compute)
• Request characteristics (expected sequence length, model required)
• Prioritization rules (e.g., VIP clients, interactive vs. batch jobs)
• Configured latency guarantees When the system is saturated, it may employ load shedding—deliberately dropping low-priority requests to preserve stability for high-priority ones.

A request queue is a buffer or data structure within the inference server where incoming client queries are held pending scheduling decisions by the orchestrator. It is the primary interface between the network layer and the batching logic. Its design directly impacts performance:

• Queue Discipline: FIFO (First-In, First-Out), Priority-based, or Deadline-aware.
• Queue Management: Parameters like batch timeout define the maximum wait time for a request, bounding latency at the cost of potentially smaller batches.
• Backpressure Mechanism: When the queue exceeds a threshold, it signals upstream clients or load balancers to slow down request submission, providing flow control.

The model scheduler is the specific sub-component of the orchestrator responsible for the temporal execution plan of model instances on hardware workers. It handles the low-level coordination required for efficient inference. Its functions include:

• Distributing batched tensors to GPU kernels
• Managing the execution of the prefill and decoding phases
• Handling operator and kernel fusion to reduce launch overhead
• Implementing iteration-level scheduling for continuous batching
• Managing KV Cache memory allocation and eviction across requests It works to minimize idle cycles on the GPU and mitigate issues like head-of-line blocking.