Evidence Score Audit Prompt for Pipeline Debugging

This prompt is for RAG pipeline operators and search engineers who are debugging unexpected evidence ranking behavior in production. Use it when a scoring model consistently overranks irrelevant passages, underranks authoritative sources, or produces scores that do not align with human judgment. The prompt takes a batch of scored passages, the original query, and the scoring configuration, then performs a structured audit to identify anomalies, miscalibrated factors, and systematic biases. It is designed for post-hoc diagnosis, not real-time scoring. Run this audit before adjusting scoring weights, retraining rankers, or modifying retrieval logic.

The ideal user has access to production scoring logs, the original query, the retrieved passage set, and the scoring model's configuration including factor weights, thresholds, and normalization parameters. You should have at least 20 scored passages to establish meaningful patterns. The prompt expects structured input: each passage must include its text, the assigned score, and ideally the per-factor breakdown if your scoring model produces one. Without factor-level detail, the audit can still detect ranking anomalies but cannot isolate which specific weight or signal is miscalibrated. Do not use this prompt for real-time scoring decisions, for evaluating a single passage in isolation, or when you lack access to the scoring configuration that produced the scores under review.

After running the audit, expect a structured output that identifies specific passages where the score appears misaligned with evidence quality, flags systematic patterns such as recency bias overwhelming relevance or authority signals being ignored, and provides root cause hypotheses with recommended remediation steps. The output is designed to feed directly into your scoring model adjustment workflow. Before acting on the audit findings, validate the identified anomalies against a small set of human judgments to confirm the audit's accuracy. Then prioritize remediation based on impact: fix systematic biases first, then address individual passage misrankings that indicate edge cases in your scoring logic.

This template is designed to be dropped directly into your debugging workflow when a retrieval-augmented generation (RAG) pipeline produces unexpected or low-quality answers. Instead of guessing why evidence selection failed, this prompt instructs the model to act as an auditor, systematically examining a batch of scored passages against the original query and the scoring criteria. The output is a structured root cause analysis, not just a new score.

textCopy
You are an evidence scoring auditor for a production RAG pipeline. Your task is to diagnose why the evidence ranking for a given query may have failed. You will receive the original user query, a list of retrieved passages with their assigned scores and the factor breakdowns for those scores, and the scoring rubric that was supposed to be applied.

Analyze the provided [SCORED_PASSAGES] against the [QUERY] and the [SCORING_RUBRIC]. Your goal is to identify anomalies, miscalibrations, and systematic biases. For each passage, determine if its score and rank are justifiable.

After your per-passage analysis, produce a root cause summary. Classify the primary failure mode into one of the following categories: `factor_miscalibration`, `source_authority_overweighted`, `recency_bias`, `specificity_undervalued`, `query_intent_mismatch`, `score_normalization_error`, or `no_anomaly_detected`.

Your final output must be a JSON object conforming to [OUTPUT_SCHEMA]. Do not include any text outside the JSON object.

[INPUT]
QUERY: [QUERY]
SCORING_RUBRIC: [SCORING_RUBRIC]
SCORED_PASSAGES: [SCORED_PASSAGES]

[OUTPUT_SCHEMA]
{
  "audit_report": {
    "query": "string",
    "per_passage_analysis": [
      {
        "passage_id": "string",
        "assigned_score": number,
        "assigned_rank": number,
        "score_justifiable": boolean,
        "anomaly_detected": boolean,
        "anomaly_description": "string | null",
        "miscalibrated_factors": ["string"]
      }
    ],
    "root_cause_analysis": {
      "primary_failure_mode": "factor_miscalibration | source_authority_overweighted | recency_bias | specificity_undervalued | query_intent_mismatch | score_normalization_error | no_anomaly_detected",
      "summary": "string",
      "impacted_passage_ids": ["string"],
      "remediation_recommendations": ["string"]
    }
  }
}

[CONSTRAINTS]
- The `miscalibrated_factors` list must only contain factor names present in the provided [SCORING_RUBRIC].
- If no anomalies are found, `primary_failure_mode` must be `no_anomaly_detected` and `impacted_passage_ids` must be an empty array.
- `remediation_recommendations` must be actionable, specific changes to the scoring prompt, weights, or normalization logic, not generic advice.

To adapt this template, start by replacing the [SCORED_PASSAGES] placeholder with a structured JSON array of your retrieved documents, each containing the passage_id, the assigned_score, and a breakdown of the factor scores that produced it. The [SCORING_RUBRIC] should be the exact text of the prompt or logic used to generate those scores, ensuring the audit can detect discrepancies between the rubric's intent and its actual output. The [OUTPUT_SCHEMA] is strict JSON; enforce this in your application layer with a schema validator before acting on the results. For high-stakes debugging, always log the full audit report alongside the original query and scores for later review.