Latency

Time to First Token is the latency from request submission until the first output token is generated by the model. This initial delay, often called 'prefill latency,' encompasses the time the system takes to process the entire input prompt, load the model weights into compute units, and perform the initial forward pass through the neural network. High TTFT is typically caused by long context lengths, cold starts, or insufficient compute for the initial prompt processing.

• Primary Driver: Prompt length and initial model computation.
• Key Optimization: Continuous batching, optimized attention mechanisms, and KV cache warm-up.

Inter-Token Latency, or time per output token, is the delay between generating successive tokens in a streaming response. After TTFT, this determines the perceived 'speed' of the output. It is governed by the incremental computation required for each new token, which is heavily dependent on model architecture size, memory bandwidth, and the efficiency of the Key-Value (KV) Cache.

• Primary Driver: Model size and memory bandwidth constraints.
• Key Optimization: Efficient KV cache management, quantization, and hardware-optimized kernels.

This component covers the time for data to travel over networks between the client, API gateways, and model-serving infrastructure. It includes:

• Round-Trip Time (RTT): The time for a request/response cycle.
• Bandwidth Limitations: Time to upload prompts and download output tokens, especially for long completions.
• Proxy/API Gateway Overhead: Processing time in intermediary routing layers.

For real-time applications like voice agents, minimizing this is critical and often necessitates edge or on-premise deployments.

For agentic systems, a significant portion of total latency can be the time spent executing tool calls to external APIs, databases, or software functions. This is often the most variable and unpredictable component.

• Examples: A weather API call (~100-500ms), a database query (~10-1000ms), or a complex software function.
• Impact: Serial tool calls are additive to total latency. Agents must be architected for parallel execution where possible.
• Monitoring: Requires detailed tool call instrumentation to attribute delays.

In multi-tenant serving systems, requests wait in a queue if all compute resources (e.g., GPUs) are busy. This delay is a function of:

• Server Concurrency: Number of requests processed simultaneously (via continuous batching).
• Request Rate vs. Throughput: Arrival rate exceeding system capacity.
• Job Scheduling Policy: How requests are prioritized (FIFO, priority-based).

High tail latency (P95, P99) is often caused by queuing under bursty traffic. Autoscaling and efficient continuous batching are key mitigations.

For systems using Retrieval-Augmented Generation (RAG) or maintaining long conversational context, latency includes the time to search and retrieve relevant information from vector databases or knowledge graphs.

• Retrieval Latency: Time for semantic search over millions of embeddings.
• Context Window Processing: Prepending retrieved documents to the prompt increases TTFT.
• Optimization: Techniques include hybrid search, pre-filtering, and optimizing embedding model inference speed.

This component shifts latency from generation to search, which can be more predictable and cacheable.

Time to First Token is the latency metric measuring the duration from when a request is sent to a generative AI model until the first token of the output stream is received by the client. This is a critical user-perceived metric for streaming applications.

• Primary Driver: Model initialization, prompt processing, and the initial computation of the output probability distribution.
• Key Distinction: While end-to-end latency measures the total time for a complete response, TTFT measures when the user first sees progress.
• Optimization Focus: Techniques like continuous batching and prefill optimization specifically target reducing TTFT.

Throughput is the rate at which an AI agent or system successfully processes requests, typically measured in requests per second (RPS) or tokens per second (TPS). It represents the system's capacity, while latency represents the delay for an individual request.

• Inverse Relationship: Under load, throughput and latency often have a trade-off. Increasing concurrent requests typically raises average latency.
• Saturation Point: The load level where latency increases exponentially while throughput plateaus, identifying the system's maximum operational capacity.
• Key for Scaling: High throughput at acceptable latency is essential for serving many users cost-effectively.

Tail latency, often expressed as the 95th (P95) or 99th (P99) percentile, measures the worst-case response times experienced by a small fraction of requests. It is critical for understanding user experience outliers and system stability.

• Focus on the Worst: While average latency is important, P95/P99 reveal the experience for the slowest 5% or 1% of requests, which users remember.
• Root Causes: Often caused by resource contention (e.g., GPU scheduling), garbage collection pauses, cold starts, or noisy neighbors in shared infrastructure.
• SLO Definition: Service Level Objectives for latency are almost always defined using P95 or P99 values, not averages.

End-to-End Latency is the total time taken for a complete user interaction with an AI agent, from the initial user input to the final, actionable output delivered back to the user. This is the holistic latency metric that matters most for task completion.

• Encompasses All Stages: Includes network round-trip, authentication, orchestration logic, tool call execution, model inference, post-processing, and response streaming.
• Agent-Specific Delays: For agentic systems, this includes time spent on planning, sequential tool execution, and reflection cycles.
• Monitoring Challenge: Requires distributed tracing across all microservices and external API calls to identify bottlenecks.

A Service Level Objective is a target value or range of values for a service level indicator (SLI) that defines the expected reliability and performance of an AI system. For latency, this is typically expressed as "X% of requests must complete under Y milliseconds."

• Latency SLO Example: "99% of agent task completions must have an end-to-end latency under 2.5 seconds (P99 < 2500ms)."
• Error Budget: The allowable amount of time the service can violate its SLO. Exhausting the budget triggers a freeze on new features to focus on stability.
• Engineering Driver: SLOs make latency a contractual, measurable requirement that guides architectural decisions and prioritization.

A Performance Bottleneck is the component or resource within an AI system that limits overall throughput or increases latency. Identifying and eliminating bottlenecks is the core activity of performance optimization.

• Common Bottlenecks in AI Systems:
  • GPU Compute: Slow model inference (prefill/generation).
  • I/O Bound: Slow vector database queries or external API calls (tool latency).
  • CPU Bound: Serialized orchestration logic or prompt rendering.
  • Network: Latency between geographically dispersed services.
• Analysis Method: Use profiling tools and distributed traces to measure the time spent in each system component.