Evidence Ranking Failure Mode Catalog Prompt

This prompt is a diagnostic tool for RAG quality and reliability engineers who need to audit their evidence ranking pipeline. It tests a set of retrieved passages against four canonical failure modes: position bias (favoring first or last passages), length bias (favoring longer passages over concise evidence), authority overfitting (overweighting source reputation at the expense of content relevance), and query misalignment (ranking passages that match keywords but miss the user's actual information need). Use this prompt when you are debugging ranking inconsistencies, calibrating a new ranker, or building a regression test suite for your retrieval pipeline. It is not a replacement for your production ranking prompt; it is a diagnostic probe that reveals where your ranking logic breaks under controlled conditions.

To use this prompt effectively, you must provide a controlled input set: a fixed query, a curated list of passages deliberately constructed to expose the four failure modes, and the ranking output from your system under test. The prompt does not rank passages itself—it analyzes the ranking decisions your system already made. This means you need to run your production ranking prompt first, capture its output, and then feed both the input passages and the resulting ranking into this diagnostic prompt. The output is a structured failure analysis report that identifies which biases are present, which passages were misranked, and how severe each failure is. This separation of concerns—ranking versus auditing—keeps the diagnostic independent of your production logic.

Do not use this prompt as your primary ranking mechanism. It is designed for offline analysis, regression testing, and pre-release evaluation, not for serving real user traffic. The prompt expects carefully constructed adversarial examples; running it on arbitrary production traffic will produce noisy, unactionable results. Instead, integrate it into your evaluation harness: run it against a golden dataset of failure-mode test cases before every ranking model or prompt update, and gate deployment on whether the failure analysis report shows regression. For high-stakes domains like medical or legal search, pair this diagnostic with human review of flagged cases before accepting the analysis as ground truth. After running the analysis, feed the identified failure patterns back into your ranking prompt design or reranking logic to close the loop.

This prompt is designed to be run as a diagnostic, not as a production ranker. Feed it a retrieval set, a query, and a reference ranking, and it will systematically probe for position bias, length bias, authority overfitting, query misalignment, and other common ranking failure modes. The output is a structured failure analysis report you can use to debug your ranking pipeline before it affects downstream answer generation.

textCopy
You are an evidence ranking diagnostic system. Your job is to analyze a set of retrieved passages against a query and a reference ranking to identify failure modes in the ranking pipeline.

## INPUTS
- Query: [QUERY]
- Retrieved Passages (unordered): [RETRIEVAL_SET]
- Reference Ranking (ground truth order): [REFERENCE_RANKING]
- Ranking Method Under Test: [RANKING_METHOD]

## FAILURE MODES TO CHECK
1. **Position Bias**: Does passage order in the input influence ranking output? Check if early-position passages receive inflated scores regardless of relevance.
2. **Length Bias**: Do longer passages receive systematically higher or lower scores? Compare score-to-length correlation.
3. **Authority Overfitting**: Are passages from high-authority sources ranked above more relevant passages from lower-authority sources? Flag cases where authority signal overrides content relevance.
4. **Query Misalignment**: Do any highly-ranked passages match query keywords but miss the underlying intent? Identify lexical matches that are semantically off-target.
5. **Redundancy Blindness**: Does the ranking surface near-duplicate passages without penalizing redundancy? Flag clusters of semantically equivalent passages occupying top slots.
6. **Coverage Collapse**: Does the top-K ranking cover all required aspects of the query, or does it over-index on one sub-topic? Check for missing query facets.
7. **Score Calibration Drift**: Are relevance scores inflated or compressed relative to the reference? Measure score distribution against reference expectations.

## OUTPUT SCHEMA
Return a JSON object with this structure:
{
  "analysis_summary": {
    "total_passages": <int>,
    "failure_modes_detected": <int>,
    "overall_severity": "low" | "medium" | "high" | "critical",
    "ranking_fidelity_score": <float 0.0-1.0>
  },
  "failure_mode_results": [
    {
      "failure_mode": <string>,
      "detected": <boolean>,
      "severity": "none" | "low" | "medium" | "high",
      "affected_passage_ids": [<string>],
      "description": <string explaining what went wrong>,
      "evidence": <string with specific examples from the ranking>,
      "recommended_action": <string>
    }
  ],
  "ranking_comparison": {
    "reference_order": [<string passage IDs>],
    "observed_order": [<string passage IDs>],
    "kendall_tau_distance": <float>,
    "top_k_overlap_at_5": <float>,
    "top_k_overlap_at_10": <float>
  },
  "calibration_analysis": {
    "mean_score": <float>,
    "score_variance": <float>,
    "score_range": [<float min>, <float max>],
    "calibration_issues": [<string>]
  }
}

## CONSTRAINTS
- Compare the observed ranking against the reference ranking, not against your own judgment of relevance.
- Flag every passage where ranking position deviates from reference by more than 2 positions.
- If the reference ranking is incomplete or ambiguous, note this as a limitation in the analysis.
- Do not re-rank the passages. Your output is a diagnostic report, not a corrected ranking.
- For each detected failure mode, provide at least one concrete example with passage IDs.

To adapt this template, replace [QUERY] with the user question that produced the retrieval set, [RETRIEVAL_SET] with the full unordered list of passages including their IDs and source metadata, [REFERENCE_RANKING] with your ground-truth ordering, and [RANKING_METHOD] with a label identifying the ranker under test (e.g., bm25, cross-encoder-v2, llm-rerank-gpt-4). Run this diagnostic before shipping ranking changes and gate deployment on the ranking_fidelity_score and overall_severity fields. For high-severity findings, fix the ranking pipeline and re-run the diagnostic before promoting to production.

This prompt is a diagnostic tool, not a production ranker. It should be wired into your offline evaluation pipeline, triggered against a golden dataset of query-evidence pairs with known failure modes. The harness must supply the prompt with a [QUERY], a [RETRIEVED_PASSAGES] list (each with an ID, text, and optional metadata like source, date, and authority score), and a [FAILURE_MODE_CATALOG] that defines the specific biases to test for—position bias, length bias, authority overfitting, and query misalignment. The output is a structured failure analysis report, not a ranked list, so the harness should parse the JSON output and compare detected failures against expected ground-truth labels in your evaluation set.

For implementation, wrap the prompt call in a function that accepts a test case object and returns a structured result. Validate the output against a strict JSON schema that requires failure_mode_detected (boolean), failure_type (enum matching your catalog), affected_passage_ids (array of strings), and explanation (string). If the model returns malformed JSON, implement a single retry with the error message injected as a [PREVIOUS_OUTPUT_ERROR] constraint. Log every invocation—including the raw prompt, model response, parse success/failure, and detected-vs-expected failure comparison—to your observability platform. For model choice, use a capable instruction-following model (GPT-4o, Claude 3.5 Sonnet, or equivalent) with temperature set to 0 for deterministic diagnostic output. Do not use this prompt in a user-facing request path; it is designed for offline analysis and should be gated behind an eval job scheduler.

The harness should also implement a coverage check: after running the prompt across your evaluation set, aggregate results to identify which failure modes are most frequently detected and which are systematically missed by your production ranker. If the prompt flags authority overfitting in 40% of test cases but your production ranker has no authority signal, that's a clear signal to add source trustworthiness weighting. Avoid the temptation to use this prompt's output as a real-time ranking correction—it is a measurement instrument, not a repair tool. The next step after running this diagnostic is to feed the failure analysis into your ranker improvement backlog, not into your request pipeline.