Evidence Ranking Error Counterexample Prompt

This prompt is for RAG system builders who need to teach a model to distinguish strong evidence from weak or irrelevant sources by showing it what incorrect prioritization looks like. The primary job-to-be-done is reducing retrieval noise and improving answer grounding by injecting negative examples that demonstrate flawed ranking decisions—favoring a document that matches keywords but contradicts the query's intent, selecting a source with low authority, or ignoring a directly relevant passage in favor of a tangentially related one. The ideal user is an AI engineer or search quality engineer who already has a working retrieval pipeline and needs to fine-tune the model's evidence selection behavior through few-shot counterexamples rather than rewriting the entire system prompt.

Use this prompt when you observe systematic ranking errors in production: the model consistently picks the first retrieved chunk regardless of relevance, gets distracted by high-term-overlap but semantically irrelevant documents, or fails to prefer authoritative sources when they are available. It is most effective when you have a labeled set of query-evidence pairs where the correct ranking is known, and you can extract specific failure cases to use as negative demonstrations. The prompt template expects you to provide [INPUT_QUERY], a [RETRIEVED_EVIDENCE_SET] with multiple candidate passages, and one or more [INCORRECT_RANKING_EXAMPLES] that show the model what not to do. Do not use this prompt when your retrieval system itself is broken—if the correct evidence is never retrieved, no amount of ranking instruction will fix the problem. This prompt addresses ranking quality, not retrieval recall.

Avoid deploying this prompt without a validation harness that measures ranking quality metrics like NDCG, MRR, or precision@k against a held-out test set. Counterexamples can over-correct: a model shown only ranking failures may become overly conservative, refusing to rank any evidence confidently or requiring excessive source verification. Pair this prompt with positive examples that show correct ranking behavior, and run regression tests to ensure the counterexamples improve ranking accuracy without introducing new failure modes. If the ranking task involves regulated content—medical, legal, or financial evidence—always include a human review step before the ranked results are surfaced to end users.

This prompt template is designed for RAG system builders who need to fine-tune a model's ability to prioritize evidence. It works by presenting a query and a set of retrieved passages, then demonstrating an incorrect ranking as a negative example. The model learns to avoid common pitfalls like prioritizing a source based on keyword density, source recency, or authority bias when the content is irrelevant to the specific query. The core mechanism is a few-shot counterexample that pairs a flawed ranking with an explicit critique, teaching the model why the ranking is wrong before it attempts the real task.

codeCopy
You are an evidence ranking assistant. Your task is to rank the provided text passages by their relevance to the user's query. The most relevant passage should be ranked first.

A good ranking directly addresses the query's core intent with specific facts. A bad ranking is distracted by superficial keyword matches, high-level topic similarity, or the perceived authority of the source when the content does not answer the question.

---
NEGATIVE EXAMPLE:

User Query: [EXAMPLE_QUERY]

Retrieved Passages:
[PASSAGE_A]: [TEXT_OF_IRRELEVANT_PASSAGE_A]
[PASSAGE_B]: [TEXT_OF_HIGHLY_RELEVANT_PASSAGE_B]
[PASSAGE_C]: [TEXT_OF_IRRELEVANT_PASSAGE_C]

Incorrect Ranking and Critique:
1. [PASSAGE_A] - INCORRECT. This passage is ranked first because it contains the keyword "[KEYWORD]" many times, but it discusses a completely different context and does not answer the query.
2. [PASSAGE_C] - INCORRECT. This passage is from a highly authoritative source, but its content is only tangentially related to the query's specific question.
3. [PASSAGE_B] - CORRECT RANK. This passage directly answers the query with specific evidence, but it was incorrectly deprioritized.

---
TASK:

Now, rank the following passages for the new user query. Provide your output as a JSON object with a "ranking" key containing an ordered list of passage IDs, and a "critique" key explaining why the top-ranked passage is the most relevant.

User Query: [INPUT_QUERY]

Retrieved Passages:
[PASSAGE_1]: [TEXT_OF_PASSAGE_1]
[PASSAGE_2]: [TEXT_OF_PASSAGE_2]
[PASSAGE_3]: [TEXT_OF_PASSAGE_3]

Output Schema:
{
  "ranking": ["PASSAGE_ID"],
  "critique": "string"
}

To adapt this template, replace the bracketed placeholders in the NEGATIVE EXAMPLE section with a realistic failure case from your own retrieval pipeline. The example must clearly show a non-relevant passage winning due to a specific, named bias (e.g., keyword density, source authority). For the TASK section, inject your live query and candidate passages. The Output Schema enforces a structured response that is easy to validate. Before deploying, run this prompt against a golden dataset where the correct ranking is known and measure the Normalized Discounted Cumulative Gain (NDCG) to ensure the counterexample is steering the model toward better precision, not just memorizing the negative case.

This prompt is not a one-shot generation task; it is a synthetic data generator for evaluation. The primary implementation pattern is to call this prompt programmatically to produce a labeled dataset of (query, evidence_list, incorrect_ranking) tuples. These tuples are then used to test your production ranker's ability to avoid the demonstrated mistakes. The harness must therefore treat the LLM call as a data factory, not a runtime component. Each generated counterexample should be validated for structural correctness before being admitted to the evaluation set.

Wiring the call: Construct the request by injecting a real [QUERY] and a set of [RETRIEVED_PASSAGES] from your retrieval system. The [INCORRECT_RANKING_PATTERN] variable should specify the type of error you want to test (e.g., 'favoring shorter passages over more relevant longer ones' or 'ranking by entity match count instead of semantic fit'). The model will return a JSON object containing the incorrect ranking order and a justification. Validation step: Before using this output, validate that the returned ranked_passage_ids array contains exactly the same IDs as the input passages, with no additions or omissions. Reject any output that fails this check and retry with a lower temperature. Log the rejection reason for quality monitoring.

Metrics integration: The primary use of this prompt's output is to compute Normalized Discounted Cumulative Gain (NDCG) and Mean Reciprocal Rank (MRR) for your production ranker against the deliberately flawed baseline. Store each generated counterexample with a unique example_id and the error_type label. Run your production ranker on the same query and passage set, then compare its NDCG against the counterexample's NDCG. A production ranker that scores lower than the counterexample on a specific error type indicates a vulnerability that needs addressing. Human review gate: For high-stakes domains (medical, legal, financial), a domain expert should review a sample of generated counterexamples to confirm that the 'incorrect' ranking is genuinely flawed and not a plausible alternative interpretation. This prevents the evaluation set from teaching the wrong lesson.

Retry and fallback: If the model fails to produce valid JSON after two retries, or if the generated ranking is identical to the ideal ranking (indicating the model failed to produce a true counterexample), discard the attempt and log the query for manual curation. Do not silently include degenerate examples in your evaluation set. Model choice: This task benefits from models with strong instruction-following and JSON output capabilities. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are suitable. Avoid smaller models that may conflate the instruction to produce an incorrect ranking with a general tendency to hallucinate rankings that don't match the input passages. Next step: After accumulating a validated counterexample dataset, use it as a regression suite—run it before every ranker update to detect ranking quality regressions on known failure patterns.