Tail Latency

Tail latency is defined by high percentile latencies (e.g., p95, p99, p99.9), not averages. The p99 latency is the time below which 99% of all requests complete. A small number of slow requests disproportionately affect these percentiles, making them essential for Service Level Objectives (SLOs).

• Average Latency (p50): Often misleading; a system can have a good average but terrible p99.
• Focus on Outliers: Engineering efforts target the worst 1% (p99) or 0.1% (p99.9) of requests.
• Example: If p99 latency is 500ms, 1 in 100 requests takes 500ms or longer.

Tail latency is caused by systemic variance and resource contention. Common technical causes include:

• Garbage Collection Pauses: Sudden stop-the-world events in managed languages (Java, Go).
• Queueing Delays: Requests waiting in a request queue due to head-of-line blocking.
• Noisy Neighbors: Contention for shared resources (CPU, network, GPU memory) in multi-tenant systems.
• Stragglers: A single slow backend instance or disk I/O operation can delay an entire batch.
• Tail-at-Scale Effects: In distributed systems, the chance of any component experiencing a delay increases with scale, amplifying the tail.

High tail latency directly degrades user-perceived performance and business metrics.

• Interactive Applications: Users notice individual slow page loads or chat responses, not averages. A 99th-percentile delay feels like a "broken" application.
• Cascading Failures: Slow requests can tie up server connections, leading to backpressure and increased error rates.
• Revenue Impact: Studies (e.g., by Amazon, Google) show direct correlation between latency increases and decreases in conversion rates and user engagement.

Combating tail latency requires techniques to reduce variance and isolate slow requests.

• Request Hedging: Sending duplicate requests to multiple replicas and using the first response.
• Fine-Grained Batching: Using continuous batching and iteration-level scheduling to prevent short requests from waiting behind long ones.
• Load Shedding & Admission Control: Rejecting requests when queues are too deep to protect latency for accepted requests.
• Prioritization & QoS: Implementing quality-of-service tiers to ensure critical requests bypass queues.
• Systematic Redundancy: Over-provisioning capacity or using cancellation to kill straggling requests.

Inference batching creates a fundamental trade-off between throughput and tail latency.

• Static Batching: Can cause severe tail latency due to head-of-line blocking; all requests wait for the slowest in the batch.
• Dynamic & Continuous Batching: Mitigates this by allowing new requests to join (iteration-level scheduling) and completed ones to exit a batch early. This is essential for maintaining low p99 latency while achieving high GPU utilization.
• The Trade-off: Larger batch sizes increase throughput but also increase the risk of a long request delaying many others, pushing out the tail.

Managing tail latency requires specific observability practices beyond average metrics.

• Histograms & Percentiles: Essential. Tools must track full latency distributions, not just averages or sums.
• High-Resolution Metrics: Capture p95, p99, p99.9 at fine time intervals (e.g., 1 second) to detect brief spikes.
• Cause Correlation: Link latency spikes to system events (deployments, GC logs, hardware errors).
• SLO/SLI Tracking: Define Service Level Indicators (SLIs) like "p99 latency < 200ms" and burn them into error budgets for automated alerting.

Head-of-line blocking is a performance degradation in batched systems where a single long-running or stalled request within a batch delays the processing and completion of all other requests in that same batch. This is a primary cause of elevated tail latency.

• Impact: In a static batch, one long sequence forces all other sequences to wait, inflating their p95/p99 latencies.
• Mitigation: Techniques like continuous batching and iteration-level scheduling allow completed sequences to exit the batch early, directly combating this effect.

Continuous batching is an inference optimization technique that dynamically groups incoming requests into batches at each model iteration. New requests can join an active batch, and completed sequences can exit, without waiting for the entire original batch to finish.

• Core Benefit: Dramatically reduces idle cycles on GPUs and minimizes the impact of head-of-line blocking, which directly lowers tail latency.
• Mechanism: Operates via iteration-level scheduling, making fine-grained decisions on which requests to process together for each token generation step.

Request admission control is a policy mechanism in an inference server that decides whether to accept, queue, or reject incoming requests based on real-time system load and capacity.

• Purpose: Prevents system overload, which is a major source of latency spikes and failures. It acts as a first line of defense for tail latency SLAs.
• Policies: Can use metrics like queue length, predicted request duration, or available GPU memory. When thresholds are breached, it may trigger load shedding (dropping requests) to protect the latency of already-accepted queries.

Percentile latency metrics, such as p95 and p99, measure the latency threshold below which a certain percentage of all requests complete. They are the standard for quantifying tail latency.

• p95 Latency: 95% of requests are faster than this value. It captures "typical" bad cases.
• p99 Latency: 99% of requests are faster than this value. It captures the extreme "tail" of the distribution, which is most critical for user-perceived performance in interactive applications.
• vs. Average: Average latency can mask outliers; p95/p99 explicitly surface them.

An inference server is a software system designed to host, manage, and serve machine learning models in production. It implements the core infrastructure that determines tail latency.

• Key Functions: Provides APIs (e.g., HTTP/gRPC), manages request queues, enforces batching policies, handles KV cache memory, and provides observability.
• Role in Latency: The server's orchestrator and scheduler components are responsible for techniques like continuous batching and request admission control, which are fundamental to controlling p95/p99 latencies.

A Service Level Objective (SLO) is a target value or range for a service-level metric, such as "p99 latency < 200ms." It is the formal contract that tail latency metrics are measured against.

• Engineering Driver: SLOs for p95/p99 latency dictate system design choices, forcing the adoption of advanced scheduling, over-provisioning, and admission control.
• Violation: Breaching an SLO indicates user experience is degrading. The strategies in this glossary (continuous batching, etc.) are engineering responses to meet stringent latency SLOs cost-effectively.