Request Admission Control

The core scheduling technique that enables efficient request admission. It dynamically groups incoming requests into a single, persistent batch that is processed iteratively.

• New requests can join the active batch as soon as the next model iteration begins.
• Completed sequences exit the batch immediately upon generating their end token, freeing resources.
• This eliminates the idle cycles inherent in static batching, dramatically improving GPU utilization and overall throughput.

The reactive counterpart to proactive admission control. When a system is critically overloaded, load shedding is the policy of deliberately dropping or rejecting requests to prevent catastrophic failure.

• Protects system stability by ensuring accepted requests can complete within their SLA.
• Often implemented with priority queues, where low-priority requests are shed first.
• Differs from admission control, which is a pre-emptive gatekeeper; load shedding is a reactive safety valve.

A flow-control mechanism that propagates system congestion signals upstream. When an inference engine's queues are full or it is overloaded, it signals to upstream components (like a load balancer or API gateway) to slow down or temporarily stop sending new requests.

• Prevents request floods from overwhelming downstream resources.
• Works in concert with admission control: admission control decides at the gate, while backpressure communicates system state to the gate.
• Essential for building resilient, cascading failure-resistant distributed systems.

The critical performance metric that admission control policies are designed to protect. Tail latency refers to the high-percentile request completion times (e.g., p95, p99).

• In inference systems, a single long-running request can cause head-of-line blocking, inflating tail latency for all others in its batch.
• Effective admission control and scheduling (like continuous batching) minimize this variance.
• Service-Level Agreements (SLAs) for interactive applications are almost always defined by tail latency, not average latency.

The data structure that holds incoming client requests pending scheduling and execution. It is the primary interface between the admission controller and the batch scheduler.

• Admission control policies often define queue discipline (e.g., FIFO, priority-based).
• A batch timeout parameter can be applied here, forcing a batch to be dispatched even if not full to bound worst-case wait time.
• Monitoring queue depth is a key signal for triggering admission control decisions and load shedding.

The set of rules that governs how the inference scheduler forms and manages batches. Admission control is one input into this policy.

• Defines heuristics for when to create a new batch (e.g., based on queue size or a time window).
• Decides which requests to select from the queue (considering sequence length, priority).
• Manages execution details like padding strategies and handling of variable-length sequences to minimize wasted computation.