Retrieval Gap Root-Cause for Eval Failure Prompt

This playbook is designed for search engineers, RAG developers, and AI reliability engineers who are investigating a specific production failure. The core job-to-be-done is to move from a symptom—an eval failure—to a structured diagnosis of the retrieval pipeline. You should use this prompt when you have access to a complete production trace, including the original user query, the final list of retrieved documents with their ranking scores, the embedding model metadata, and the specific eval rubric that the response failed. The ideal user is someone who can query their observability platform, extract these trace segments, and format them as structured inputs for the prompt.

The prompt works by systematically comparing the evidence required for a passing eval against the evidence that was actually retrieved. It isolates the failure to one or more specific stages: query formulation (was the search query a poor representation of user intent?), embedding mismatch (did the query and documents live in different semantic spaces?), index staleness (was the correct information not yet indexed?), ranking cutoff (was the relevant document retrieved but ranked below the top-K threshold?), or filter misconfiguration (did a metadata filter incorrectly exclude the right documents?). For example, if an eval requires a specific regulatory clause to be cited and the trace shows the clause exists in the index but was ranked 15th with a top-K of 10, the prompt will flag a ranking cutoff issue rather than an index staleness issue.

Do not use this prompt when the eval failure is clearly caused by generation hallucination, tool-call errors, or prompt instruction violations without retrieval involvement. If the model had the correct documents in its context window but still produced an incorrect or unsupported answer, the root cause is in the generation layer, not the retrieval layer. In such cases, you should use the sibling playbooks for hallucination spot-checking or instruction leakage analysis. After running this diagnosis, the next step is to take the structured output—which includes a root-cause category, affected documents, and a confidence score—and feed it into your retrieval improvement workflow, whether that means tuning the embedding model, adjusting the ranker, or updating the query rewriter.

This prompt template is designed to be copied directly into your investigation environment. It forces a structured comparison between the query intent, the retrieval pipeline's actual behavior, and the evidence required to pass the failed evaluation. Before using it, gather the failed eval's scorecard, the full trace from the production session (including embedding vectors, retrieval scores, and ranking metadata), and the source documents or index snapshot from the time of the request. The prompt assumes you have access to structured trace data—if your observability platform only captures final outputs, you will need to enrich your tracing before this prompt can produce a reliable diagnosis.

textCopy
You are a retrieval quality investigator analyzing a production RAG session that failed an automated evaluation. Your task is to identify the root cause of the retrieval gap that led to the eval failure.

## INPUT DATA

### Failed Eval Session
- **Eval Rubric:** [EVAL_RUBRIC_DESCRIPTION]
- **Failed Dimension(s):** [FAILED_EVAL_DIMENSIONS]
- **Eval Score:** [EVAL_SCORE] / [MAX_SCORE]
- **Eval Judge's Explanation:** [EVAL_JUDGE_EXPLANATION]

### User Query

[USER_QUERY]

codeCopy

### Retrieval Pipeline Trace
- **Query Rewriting Applied:** [REWRITTEN_QUERIES]
- **Embedding Model:** [EMBEDDING_MODEL_NAME]
- **Index/Collection:** [INDEX_NAME]
- **Retrieval Method:** [RETRIEVAL_METHOD]
- **Top-K Requested:** [TOP_K]
- **Retrieved Documents (with scores and ranks):**
```json
[RETRIEVED_DOCUMENTS_ARRAY]

• Reranking Applied: [RERANKING_DETAILS]
• Final Context Window Contents (truncated to what the model saw):

codeCopy
[CONTEXT_WINDOW_CONTENTS]

Generated Output

codeCopy
[GENERATED_OUTPUT]

Expected Evidence for Passing Eval

• Required Facts or Citations: [REQUIRED_EVIDENCE_LIST]
• Source Documents That Should Have Been Retrieved: [EXPECTED_SOURCE_DOCUMENTS]

INVESTIGATION INSTRUCTIONS

1. Reconstruct the retrieval intent: What information did the user query actually need to produce a passing answer? Compare this against the rewritten queries actually sent to the retriever.

2. Audit the retrieval results: For each required piece of evidence, determine:

   • Was it present in the index at request time?
   • If present, what was its retrieval rank and score?
   • If absent, is this an indexing gap (document missing from index) or a retrieval gap (document present but not surfaced)?

3. Diagnose the gap mechanism. Classify the root cause into one or more of:

   • Query formulation failure: The rewritten queries did not capture the semantic or lexical intent needed to surface the evidence.
   • Embedding mismatch: The query embedding and document embeddings are not aligned for this type of content (e.g., domain-specific terminology, code, or structured data).
   • Ranking failure: The evidence was retrieved but ranked below the top-K cutoff or dropped during reranking.
   • Context window truncation: The evidence was retrieved and ranked high enough, but was truncated from the final context window due to token limits.
   • Index staleness: The required evidence was added to the source after the last index update.
   • Metadata filter error: A metadata filter excluded the relevant documents unintentionally.

4. Quantify the gap: For each missing piece of evidence, state its retrieval rank (if retrieved), its relevance score, and the score threshold of the lowest-ranked document that made it into the context window.

5. Assess downstream impact: Explain how each retrieval gap directly caused the specific eval dimension to fail. Connect the missing evidence to the eval judge's explanation.

OUTPUT FORMAT

Return a JSON object with this schema:

jsonCopy
{
  "retrieval_intent_summary": "string",
  "query_formulation_analysis": {
    "original_query": "string",
    "rewritten_queries": ["string"],
    "gap_assessment": "string",
    "suggested_query_fix": "string or null"
  },
  "evidence_audit": [
    {
      "required_evidence": "string",
      "present_in_index": true,
      "retrieval_rank": 12,
      "retrieval_score": 0.67,
      "made_it_to_context_window": false,
      "failure_category": "ranking_failure",
      "explanation": "string"
    }
  ],
  "root_cause_classification": {
    "primary_cause": "ranking_failure",
    "contributing_causes": ["context_window_truncation"],
    "confidence": "high",
    "evidence_summary": "string"
  },
  "eval_failure_linkage": {
    "failed_dimension": "string",
    "causal_chain": "string",
    "severity": "critical"
  },
  "remediation_recommendations": [
    {
      "action": "string",
      "target_component": "string",
      "expected_impact": "string",
      "effort_estimate": "low"
    }
  ]
}

CONSTRAINTS

• Do not speculate about index contents you cannot see. If the trace does not include full index state, flag this as a limitation.
• Distinguish clearly between "evidence was not in the index" and "evidence was in the index but not retrieved."
• If the eval failure is not actually caused by a retrieval gap, state this explicitly and suggest alternative root causes.
• Cite specific retrieval ranks, scores, and token positions from the trace data.

Adapt this template by replacing each square-bracket placeholder with actual trace data. The [RETRIEVED_DOCUMENTS_ARRAY] placeholder expects a JSON array of objects containing at minimum doc_id, content_snippet, retrieval_score, and rank. If your tracing system captures embedding vectors, include them in the [EMBEDDING_MODEL_NAME] context but do not paste raw vectors into the prompt—they consume tokens without adding reasoning value. The [EXPECTED_SOURCE_DOCUMENTS] field is critical: you must manually identify which documents or facts were needed to pass the eval. If you cannot determine this, run the eval rubric against a known-good answer first to extract the required evidence set. For high-stakes investigations where the remediation will change production retrieval behavior, route the output through a human review step before applying changes to query rewriting, embedding models, or index configuration.

This prompt is designed to be triggered automatically when a RAG eval failure is traced to a retrieval gap—specifically, when the faithfulness or grounding score drops below a threshold and the trace shows that the required evidence was absent from the retrieved context. The harness should receive structured trace data as input, not raw logs. You'll need to extract the user query, the final model response, the eval rubric and failing score, the list of retrieved document IDs with their ranks and scores, and the embedding model version from your observability platform before calling this prompt.

Wire the prompt into your incident pipeline as a diagnostic step that runs after eval failure classification but before human triage. The input should be assembled from your trace store (e.g., LangSmith, Arize, or a custom trace database) and passed as a structured JSON object. Validate the input schema before calling the model: confirm that retrieved_documents is non-empty, that eval_rubric contains the failing dimension, and that query and generated_response are present. If any required field is missing, log a schema validation error and escalate to the on-call engineer rather than running the prompt with incomplete data. The output should be parsed as JSON and validated against an expected schema with fields for root_cause_category, evidence_gap_summary, pipeline_stage_identified, and recommended_action. If the model returns malformed JSON, apply a repair prompt or retry once before falling back to human review.

For continuous monitoring, run this prompt on a sampled subset of retrieval-gap eval failures—not every single event—to control cost and latency. Batch the results into a daily diagnostic digest that tracks the most frequent root-cause categories (e.g., query_rewrite_failure, embedding_drift, index_staleness, ranking_cutoff). Use these aggregates to trigger automated alerts when a category exceeds a threshold (e.g., more than 10% of sampled failures attributed to embedding drift within 24 hours). Do not use this prompt for real-time request correction; it is a diagnostic tool for post-hoc root-cause analysis. The recommended actions it produces should feed into your prompt improvement backlog, index refresh cadence, or embedding model migration plan—not into live request rewriting without human approval.