End-to-End Latency

This initial phase encompasses the time to parse and understand the user's natural language query before retrieval begins. It includes operations like:

• Spelling correction and query normalization.
• Intent classification to determine the user's goal.
• Query expansion or rewriting to improve retrieval recall.
• Embedding generation for the query vector in dense retrieval systems. Latency here is typically low (tens to low hundreds of milliseconds) but can spike with complex linguistic analysis or large embedding models.

This is the core retrieval operation where the system searches a knowledge base for relevant context. Latency is dominated by:

• Index traversal in a vector database (e.g., approximate nearest neighbor search via HNSW or IVF).
• Hybrid search overhead, combining dense vector similarity with sparse keyword (e.g., BM25) scores.
• Network I/O between the application and the database cluster. Performance scales with index size, embedding dimensionality, and the chosen similarity search algorithm's trade-off between speed and recall.

After retrieving a broad set of candidates (e.g., top 100), a secondary, more precise model reranks them. This phase adds latency but significantly improves answer quality. It involves:

• Running a cross-encoder model (like a MiniLM) to score query-document pairs.
• Applying metadata filters (date, source) or heuristic rules.
• Selecting the final top-K passages for the context window. While powerful, cross-encoders are computationally expensive, making this a common bottleneck for low-latency applications.

The longest and most variable phase, where the language model consumes the retrieved context to generate an answer. Latency is determined by:

• Prompt templating and context window stuffing.
• LLM inference time, which scales with output token count, model size, and the decoding method (e.g., greedy vs. beam search).
• Network latency to the model endpoint (API or self-hosted).
• Speculative decoding or caching can reduce time-to-first-token and total generation time.

The final step before returning the answer to the user, often performed in parallel with generation streaming. This includes:

• Answer validation or formatting against a schema.
• Citation anchoring: linking specific statements in the generated text back to source document chunks.
• Hallucination checks or safety filtering.
• Structured output generation (JSON, XML). While often lightweight, complex post-processing logic or external verification calls can add measurable delay.

The foundational latency not attributable to a specific RAG task, but inherent to the production system. This includes:

• Network latency between user, application servers, and downstream services (databases, LLM APIs).
• Load balancer routing and API gateway processing.
• Request queuing under high load, especially for GPU-bound tasks like inference and reranking.
• Serialization/deserialization of data (e.g., Protobuf/JSON). Optimizing this requires systems engineering focus on architecture, scaling, and concurrency models.

The time added by applying a secondary, more computationally intensive model (a cross-encoder or a LLM judge) to reorder an initial set of retrieved documents for improved relevance.

• Trade-off: Rerankers like Cohere or Sentence Transformers cross-encoders significantly improve Retrieval Precision and NDCG but introduce a sequential processing delay.
• Impact: This stage is often the bottleneck for latency in high-precision RAG systems, as it processes every retrieved candidate document.

The time the large language model takes to produce the final answer conditioned on the retrieved context. This is typically the most variable and resource-intensive phase.

• Drivers: Model size (e.g., 7B vs. 70B parameters), decoding method (greedy vs. beam search), output token length, and inference hardware (GPU/TPU).
• Reduction Techniques: Methods include continuous batching, speculative decoding, and model quantization to improve tokens-per-second throughput.

A critical sub-component of generation latency, measuring the delay from when the generation request is sent until the first token of the output is received.

• User Perception: TTFT is crucial for perceived responsiveness in streaming interfaces.
• Influenced By: Model loading, prompt processing (prefill), and system queueing. High TTFT can indicate issues with batch size or insufficient compute resources.

A quality metric that evaluates whether a generated answer is factually consistent with and fully supported by the provided source context. It directly opposes the Hallucination Rate.

• Relationship to Latency: Systems optimized for low latency may shortcut thorough context grounding, increasing hallucination risk. Evaluating faithfulness ensures speed gains don't compromise factual accuracy.
• Measurement: Often scored by asking an LLM judge if all statements in the answer can be inferred from the context.

A target for system reliability or performance that is agreed upon with users. For RAG, an SLO often defines the maximum acceptable End-to-End Latency (e.g., p95 latency < 2 seconds) alongside quality targets like a minimum Answer Faithfulness score.

• Engineering Focus: SLOs force trade-off decisions between speed, cost, and accuracy, guiding infrastructure scaling and model selection.
• Monitoring: Requires robust Latency Benchmarking and telemetry to track violations and error budgets.