Latency Contribution by Retrieval Step Prompt

Use this prompt when you need to attribute end-to-end latency in a RAG pipeline to specific retrieval substeps—embedding generation, vector search, reranking, and context assembly—separate from model inference time. This is essential for infrastructure engineers and RAG developers who suspect that retrieval, not generation, is the dominant latency contributor but lack a structured breakdown to prove it. The prompt expects a production trace span that includes timestamps for each retrieval substep alongside the final generation span. Without per-step timing data, the prompt cannot produce a meaningful breakdown, so verify that your observability instrumentation captures these substeps before using this playbook.

The prompt produces a latency attribution table with absolute and percentage contributions for each retrieval substep, a comparison against the generation latency, and anomaly flags when any substep exceeds a configurable threshold. You must provide the trace data in a structured format, including span names, start and end timestamps, and any metadata about the retrieval configuration (e.g., embedding model, index type, reranker). The output is designed to be consumed by a monitoring dashboard or incident report, not as a conversational summary. Wire the prompt into an automated trace analysis pipeline where traces exceeding an end-to-end latency budget trigger this breakdown automatically, rather than running it ad-hoc on every request.

Do not use this prompt when the trace lacks substep granularity, when the retrieval pipeline is a single opaque API call, or when you only need a coarse binary answer about whether retrieval is slow. In those cases, a simpler per-step latency comparison prompt is more appropriate. Also avoid this prompt when the generation step dominates latency and retrieval optimization is not the immediate priority—running it on every slow trace without filtering wastes token budget. If the trace includes tool calls or multi-model routing, use the broader per-step latency breakdown prompt instead, as this prompt is specialized for retrieval-only pipelines. After identifying the bottleneck substep, follow up with targeted optimization work on that component—embedding model selection, index tuning, reranker latency reduction, or context assembly refactoring—rather than treating the breakdown as the final deliverable.

This section provides a copy-ready prompt template you can drop into your observability tooling, internal debugging notebook, or automated trace analysis pipeline. The prompt is designed to accept a structured trace span from a RAG pipeline and produce a latency attribution breakdown that separates retrieval work (embedding generation, vector search, reranking, context assembly) from model inference time. Use this template when you have end-to-end trace data and need to determine whether retrieval or generation is the dominant latency contributor before making infrastructure or configuration changes.

textCopy
You are a performance analysis assistant specialized in RAG pipeline latency diagnosis. Your task is to analyze a production trace span and produce a latency attribution breakdown that isolates retrieval steps from generation steps.

## INPUT
You will receive a structured trace object containing span timestamps, durations, and metadata for a single RAG request. The trace includes the following fields:

[TRACE_JSON]

## TASK
1. Parse the trace spans and identify each distinct step in the request lifecycle.
2. Classify each step into one of the following categories:
   - EMBEDDING_GENERATION: Time spent creating embeddings for the query or documents.
   - VECTOR_SEARCH: Time spent executing the vector similarity search.
   - RERANKING: Time spent reranking retrieved candidates.
   - CONTEXT_ASSEMBLY: Time spent formatting, truncating, or assembling retrieved context for the prompt.
   - MODEL_INFERENCE: Time spent waiting for the LLM to generate tokens.
   - OVERHEAD: Any remaining time not attributable to the above categories (network, serialization, queue wait).
3. Calculate the total duration and percentage contribution for each category.
4. Flag any step that exceeds its expected latency threshold. Use the following default thresholds unless overridden:
   - EMBEDDING_GENERATION: [EMBEDDING_THRESHOLD_MS]
   - VECTOR_SEARCH: [VECTOR_SEARCH_THRESHOLD_MS]
   - RERANKING: [RERANKING_THRESHOLD_MS]
   - CONTEXT_ASSEMBLY: [CONTEXT_ASSEMBLY_THRESHOLD_MS]
   - MODEL_INFERENCE: [INFERENCE_THRESHOLD_MS]
5. If the trace contains multiple retrieval calls (e.g., hybrid search), break down retrieval latency by each retrieval sub-step.

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "trace_id": "string",
  "total_duration_ms": number,
  "breakdown": [
    {
      "category": "EMBEDDING_GENERATION | VECTOR_SEARCH | RERANKING | CONTEXT_ASSEMBLY | MODEL_INFERENCE | OVERHEAD",
      "duration_ms": number,
      "percentage": number,
      "sub_steps": [
        {
          "step_name": "string",
          "duration_ms": number,
          "threshold_exceeded": boolean,
          "threshold_ms": number
        }
      ]
    }
  ],
  "anomalies": [
    {
      "step_name": "string",
      "observed_ms": number,
      "expected_ms": number,
      "severity": "LOW | MEDIUM | HIGH",
      "hypothesis": "string"
    }
  ],
  "retrieval_total_ms": number,
  "generation_total_ms": number,
  "retrieval_percentage": number,
  "generation_percentage": number
}

## CONSTRAINTS
- Do not invent durations or steps not present in the trace.
- If a step's duration cannot be determined from the trace data, mark it as "UNKNOWN" and exclude it from percentage calculations.
- If the trace contains nested or overlapping spans, resolve parent-child relationships before calculating totals to avoid double-counting.
- Flag anomalies only when the observed duration exceeds the threshold by 50% or more.
- If no thresholds are provided, use the trace's own p50 durations from [BASELINE_TRACE_JSON] as the expected values.
- Return valid JSON only. No markdown, no commentary outside the JSON object.

## EXAMPLES
[FEW_SHOT_EXAMPLES]

To adapt this template, replace the square-bracket placeholders with your actual trace data and configuration. The [TRACE_JSON] placeholder expects a structured trace object—this should come from your observability backend (e.g., LangSmith, Arize, Datadog, or a custom trace store). The threshold placeholders ([EMBEDDING_THRESHOLD_MS], etc.) let you define per-step latency budgets; if you omit them, the prompt falls back to comparing against a baseline trace provided in [BASELINE_TRACE_JSON]. The [FEW_SHOT_EXAMPLES] placeholder is optional but strongly recommended for production use—include one or two annotated trace-to-breakdown examples to stabilize the output format and anomaly detection logic. Before wiring this into an automated pipeline, validate the output JSON against the schema using a strict parser, and log any parse failures for human review. For high-stakes latency investigations (e.g., SLO violations affecting paying customers), always pair this prompt's output with raw trace visualizations and do not rely on the model's anomaly hypotheses without engineering confirmation.

This prompt is designed to be called programmatically, not pasted into a chat window. The primary integration point is your existing observability stack—Datadog, Grafana, Honeycomb, or a custom trace store. The harness should extract the relevant trace spans for a single request, serialize them into the [TRACE_JSON] placeholder, and invoke the model. Because latency analysis is time-sensitive during incidents, the harness must enforce a strict timeout on the model call (e.g., 15 seconds) and fall back to a simpler heuristic if the model doesn't return in time. Do not run this prompt on every request; target it at traces that breach a latency SLO threshold or are explicitly flagged by an on-call engineer. The output schema—a structured JSON breakdown of retrieval vs. generation latency—is designed to be ingested by a dashboard widget or incident response bot, so your harness must validate the JSON structure before forwarding it downstream.

Validation is the critical gate here. Before accepting the model's output, validate that the sum of the reported sub-step latencies (embedding, vector search, reranking, assembly) does not exceed the total retrieval latency, and that the total retrieval plus generation latency matches the end-to-end trace duration within a 5% tolerance. If validation fails, retry once with a simplified prompt that omits the [BASELINE_LATENCY_BUDGET] and asks only for the raw breakdown. Log both the raw trace and the model's output to your observability platform, indexed by trace ID, so you can audit for hallucinated latency numbers later. For model choice, use a low-latency model like claude-3-haiku or gpt-4o-mini because this is a structured extraction task, not a reasoning-heavy one. The prompt's [CONSTRAINTS] block should include a hard limit on output tokens (e.g., 300) to prevent verbose explanations from adding latency to your diagnostic loop.

Wire the validated output into your alerting and dashboarding layer. If the retrieval step accounts for more than 40% of the end-to-end latency budget, trigger a warning that routes to the search infrastructure team. If reranking alone exceeds 100ms, flag it for the model serving team. Avoid the temptation to run this prompt on every slow trace without sampling; batch analysis of historical traces should use a separate, cost-optimized pipeline. The immediate next step after implementing this harness is to create a runbook that maps each latency anomaly pattern to the responsible team and remediation action, so the prompt's output drives action rather than just generating a report.