Retrieval Source Ranking Audit Prompt

This prompt is a diagnostic instrument for search engineers and RAG developers who need to audit the ranked order of documents retrieved in a production trace. Its primary job is to assess whether the system's ranking reflects true relevance to the user's query, flag misranked sources, and compare the system's ranking against available relevance labels or user click data. Use it when you suspect ranking degradation after a model or index update, need to calibrate a new reranker, or want to quantify ranking quality against a ground-truth signal before shipping a retrieval change.

The ideal user brings a specific production trace containing the user query, the ordered list of retrieved document IDs or snippets, and an optional ground-truth signal such as graded relevance labels (e.g., 0-3 scale) or user click data. The prompt works offline on a single trace at a time and is not designed for real-time retrieval decisions. It expects structured input: a query string, a ranked list of sources with their content or summaries, and any available relevance judgments. Without ground-truth labels, the prompt performs a qualitative assessment of ranking coherence; with labels, it can compute rank-aware metrics like NDCG or flag specific inversion pairs where a less relevant document outranks a more relevant one.

Do not use this prompt for real-time retrieval decisions, as it is a diagnostic tool for offline trace analysis. It is also not a replacement for full retrieval evaluation frameworks that compute aggregate metrics across large query sets. If you need to evaluate retrieval quality across thousands of traces, use this prompt to calibrate your understanding on representative samples, then implement programmatic metric computation. The prompt's value lies in producing human-readable explanations of ranking failures—specific inversion pairs, relevance misjudgments, and patterns that automated metrics alone might obscure.

The following prompt template is designed to be dropped directly into your observability stack, an LLM evaluation harness, or a manual trace review workflow. It instructs the model to act as a search quality auditor, systematically comparing the ranked order of retrieved documents against a set of relevance labels or user click data. The goal is to produce a structured audit report that flags misranked sources, identifies ranking inversions, and quantifies the gap between the system's ranking and the ideal order.

textCopy
You are a search quality auditor reviewing a production retrieval trace. Your task is to audit the ranked order of retrieved documents and assess whether the ranking reflects true relevance to the user query.

## INPUTS
- User Query: [USER_QUERY]
- Ranked Retrieved Documents (in order): [RANKED_DOCUMENTS]
- Relevance Labels or Click Data: [RELEVANCE_DATA]

## INSTRUCTIONS
1. Parse the [RANKED_DOCUMENTS] list and the [RELEVANCE_DATA] to understand the ideal relevance order.
2. Compare the actual ranking against the ideal ranking derived from [RELEVANCE_DATA].
3. Identify and flag every instance where a less relevant document is ranked above a more relevant one (a ranking inversion).
4. For each flagged inversion, explain why the documents are misordered and estimate the impact on answer quality.
5. Calculate a ranking quality score using Normalized Discounted Cumulative Gain (NDCG) or Mean Reciprocal Rank (MRR), as specified in [METRIC].
6. Summarize the overall ranking health and highlight the top 3 most damaging misrankings.

## OUTPUT_SCHEMA
Return a single JSON object with the following structure:
{
  "query": "string",
  "ranking_quality_score": number,
  "metric_used": "[METRIC]",
  "total_documents_retrieved": number,
  "inversions_found": number,
  "inversions": [
    {
      "rank_position": number,
      "document_id": "string",
      "expected_rank": number,
      "explanation": "string",
      "impact": "low|medium|high"
    }
  ],
  "top_damaging_misrankings": [
    {
      "document_id": "string",
      "current_rank": number,
      "expected_rank": number,
      "reason": "string"
    }
  ],
  "overall_assessment": "string"
}

## CONSTRAINTS
- Base your analysis strictly on the provided [RELEVANCE_DATA]. Do not infer relevance from document content alone.
- If [RELEVANCE_DATA] is incomplete or missing for some documents, note this in the `overall_assessment` and exclude those documents from score calculation.
- Do not hallucinate document content. Only reference document IDs and the provided snippets.

To adapt this template for your production pipeline, replace the square-bracket placeholders with data extracted from your trace logs. The [RANKED_DOCUMENTS] should be a serialized list of document objects containing at minimum an id and a snippet. The [RELEVANCE_DATA] can be human-labeled relevance scores, implicit click-through signals, or a graded relevance scale. For high-stakes pipelines where ranking errors directly impact user trust, route the output of this prompt to a human reviewer before accepting any automated ranking score as a release gate. Always log the raw audit report alongside the trace for future regression comparison.

This prompt is designed to operate as a post-hoc trace analysis step, not a real-time ranking service. Wire it into an offline audit pipeline that consumes production traces from your observability store (e.g., LangSmith, Arize, Datadog LLM, or a custom trace database). The typical integration pattern is: a scheduled job or event-driven trigger pulls a batch of traces where retrieval ranking is suspected to be suboptimal—flagged by low user feedback scores, high bounce rates, or manual QA sampling—and sends each trace through this prompt for structured ranking assessment.

The implementation harness requires several concrete components. Input assembly: extract the user query, the ordered list of retrieved document IDs and snippets, and any available relevance labels or click data from the trace. Map these into the [QUERY], [RANKED_DOCUMENTS], and [RELEVANCE_SIGNALS] placeholders. Validation layer: after the model returns its JSON output, validate that every document in the input appears in the output ranking, that no hallucinated document IDs are introduced, and that the misranked_sources array only references documents present in the original list. If validation fails, retry once with the validation errors appended to the prompt as [PREVIOUS_ERRORS]. Logging: store the full prompt, response, validation result, and any retry attempts back into your trace store, keyed to the original trace ID. This creates an auditable record of every ranking audit.

For high-stakes retrieval pipelines—such as those serving regulated industries, medical information, or financial data—add a human review gate before accepting the audit's conclusions. Route traces where the prompt flags severity: "high" misrankings or where the ranking_quality_score falls below a configurable threshold (e.g., 0.7) to a review queue. The human reviewer should confirm or reject the flagged misrankings, and their decisions should be logged as ground-truth labels to calibrate future automated audits. Model choice: use a model with strong instruction-following and JSON output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet). Avoid smaller or older models that may struggle with the structured comparison task. Cost control: batch multiple trace audits into a single job, but keep each prompt invocation scoped to one trace to avoid context-window overflow when document lists are large. If a single trace contains more than 50 retrieved documents, split the ranking audit into overlapping windows and merge results with a tie-breaking pass.