Retrieval Query Rewrite Trace Prompt

This prompt is designed for search engineers and RAG developers who need to perform a surgical diagnosis on a single production trace to determine whether a query rewriting step improved or degraded retrieval quality. The core job-to-be-done is comparing the retrieval precision of the original user query against the rewritten queries. You should use this prompt when you have captured a complete trace in your observability platform that includes the original user query, the set of rewritten queries generated by your system, the final list of retrieved document chunks with their ranking scores, and the final generated answer. The prompt's value is in isolating the rewrite step's specific contribution to the overall outcome, separating it from generation or ranking issues.

To use this prompt effectively, you must provide a structured trace object. The required inputs are: the [ORIGINAL_QUERY], a list of [REWRITTEN_QUERIES], a list of [RETRIEVED_CHUNKS] with their associated scores, and the [FINAL_ANSWER]. The prompt is designed to produce a structured assessment that identifies specific failure modes such as synonym drift (where a rewrite term has a different meaning in the target domain), over-expansion (where the rewrite broadens the query so much that it retrieves irrelevant results), or missed intent (where a critical constraint from the original query is dropped). The output is not a simple pass/fail; it's a qualitative diagnostic report with actionable recommendations for adjusting your rewrite prompt or strategy.

Do not use this prompt for aggregate pipeline monitoring, real-time alerting, or as a replacement for offline evaluation metrics like Mean Reciprocal Rank (MRR) or Normalized Discounted Cumulative Gain (NDCG). It is not designed to process batches of traces or to trigger automated rollbacks. Its strength is in isolated, human-in-the-loop trace inspection during specific workflows: incident response, where a retrieval failure has been escalated; QA review, where you are manually auditing a sample of production sessions; or rewrite-prompt iteration, where you are A/B testing a new rewriting strategy and need to understand why one variant underperformed on a specific case. For continuous monitoring, you should implement a separate evaluation harness that uses this prompt's rubric as a scoring function, but the prompt itself is a manual diagnostic tool.

This prompt is designed to be pasted directly into your LLM playground, internal diagnostics console, or trace review tool. It instructs the model to act as a search quality analyst, comparing the original user query against the rewritten queries, the documents they retrieved, and the final answer generated. The goal is to produce a structured assessment of whether the rewrite improved, degraded, or had no effect on retrieval precision and answer quality. Before execution, you must replace every square-bracket placeholder with actual data extracted from your production trace spans.

textCopy
You are a search quality analyst reviewing a single Retrieval-Augmented Generation (RAG) trace. Your task is to evaluate the effectiveness of a query rewriting step by comparing the original user query against the rewritten queries, the documents each query retrieved, and the final answer produced.

## INPUT DATA

### Original User Query
[ORIGINAL_QUERY]

### Rewritten Queries
[REWRITTEN_QUERIES]

### Documents Retrieved by Original Query
[ORIGINAL_RETRIEVED_DOCUMENTS]

### Documents Retrieved by Rewritten Queries
[REWRITTEN_RETRIEVED_DOCUMENTS]

### Final Answer Generated
[FINAL_ANSWER]

### Expected or Ground Truth Answer (Optional)
[GROUND_TRUTH_ANSWER]

## ANALYSIS INSTRUCTIONS

1.  **Intent Preservation**: Did the rewritten queries preserve the core intent, entities, and constraints of the original query? Note any intent drift, over-specification, or critical omissions.
2.  **Retrieval Precision Comparison**: Compare the relevance of documents retrieved by the original query versus the rewritten queries. Identify if the rewrite surfaced more relevant documents, introduced more noise, or missed key documents that the original query found.
3.  **Answer Quality Impact**: Trace how the difference in retrieved documents affected the final answer. Did the rewrite enable a more accurate, complete, or well-cited answer, or did it cause a hallucination or omission?
4.  **Rewrite Strategy Assessment**: Classify the rewrite strategy used (e.g., decomposition, synonym expansion, abstraction, metadata filtering) and judge if it was the appropriate strategy for this query type.

## OUTPUT FORMAT

You must respond with a valid JSON object conforming to this schema:
{
  "rewrite_quality_assessment": {
    "intent_preserved": boolean,
    "intent_drift_notes": "string explaining any drift detected",
    "retrieval_precision_change": "improved" | "degraded" | "no_change",
    "retrieval_comparison_details": "string comparing key documents found or missed",
    "answer_quality_impact": "positive" | "negative" | "neutral",
    "answer_impact_details": "string linking document changes to answer changes",
    "rewrite_strategy_used": "string describing the strategy",
    "strategy_appropriateness": "appropriate" | "inappropriate" | "uncertain",
    "overall_verdict": "rewrite_helped" | "rewrite_hurt" | "rewrite_neutral",
    "recommendation": "string with actionable advice for the prompt engineer"
  }
}

## CONSTRAINTS
- Base your analysis strictly on the provided trace data. Do not invent documents or queries.
- If the final answer contains a hallucination not supported by any retrieved document, you must flag it in the `answer_impact_details`.
- If the [GROUND_TRUTH_ANSWER] is provided, use it to validate the final answer's correctness, but do not penalize the rewrite for retrieval gaps if the ground truth was not in the document store.

To adapt this template, replace the placeholders with raw trace data. For [ORIGINAL_RETRIEVED_DOCUMENTS] and [REWRITTEN_RETRIEVED_DOCUMENTS], include the document ID, rank, and a snippet of the text. If your trace does not capture what the original query would have retrieved in isolation, you may need to run a separate retrieval step to populate that field. The [GROUND_TRUTH_ANSWER] is optional but highly recommended for high-stakes evaluations; when omitted, the model will rely solely on internal consistency and document relevance. After generating the JSON output, you should validate it against the schema programmatically before accepting the assessment. For regulated or customer-facing search applications, always have a human review the intent_drift_notes and recommendation before changing the rewrite prompt in production.

This prompt is designed to be a single step in a larger trace review harness, not a standalone chat. The harness should extract the required inputs—the original user query, the rewritten queries, the retrieved results, and the final answer—from your observability platform's trace data before invoking the model. Because the prompt's value depends on complete input data, the harness must first validate that all required trace segments are present. If the trace is missing the rewritten query or the retrieval payload, the harness should abort and flag the trace as incomplete rather than running the prompt on partial data.

The implementation should follow a structured pipeline: Extract → Validate → Prompt → Parse → Evaluate. In the Extract phase, pull the relevant spans from your tracing system (e.g., LangSmith, Arize, Datadog, or custom trace store) using the trace ID. In the Validate phase, confirm that the original query, rewritten query list, retrieved document set with scores, and final generated answer are all non-null. If validation fails, log a trace_incomplete event and skip the prompt. In the Prompt phase, populate the template's placeholders with the extracted data and send it to a capable model (GPT-4o, Claude 3.5 Sonnet, or equivalent). In the Parse phase, enforce the output schema—the model must return valid JSON matching the RewriteQualityAssessment structure. If parsing fails, retry once with a stricter schema instruction; if it fails again, log a parse_failure and escalate for human review. In the Evaluate phase, run automated checks: verify that the retrieval_precision_before and retrieval_precision_after fields are numeric and within 0–1, that the rewrite_quality_score is an integer 1–5, and that the failure_mode field is not null when the score is below 3.

For production use, wrap this prompt in a versioned function with structured logging. Log the trace ID, the model version used, the prompt template version, the raw model output, the parsed assessment, and any parse or validation errors. This log becomes the audit trail for query rewrite quality over time. If you're running this prompt across many traces for aggregate analysis, batch the evaluations and write the structured outputs to a database or metrics store. Avoid the temptation to use this prompt for real-time query rewriting during user requests—it is a diagnostic tool for offline trace review, not a runtime rewriter. The next step after implementing this harness is to connect the structured output to a dashboard that tracks rewrite quality trends, retrieval precision deltas, and failure mode distributions across your production traffic.