Evidence Sufficiency Scorecard Prompt

This prompt is designed for AI quality teams and RAG operations engineers who need a structured, multi-dimensional assessment of whether retrieved evidence is sufficient to answer a user query. It produces a scorecard that evaluates evidence across five dimensions: relevance, completeness, authority, recency, and specificity. The output is suitable for ingestion into monitoring dashboards, SLA tracking, and continuous retrieval quality improvement workflows. Use this prompt when you need a repeatable, calibrated method to score retrieval sets before they reach answer generation.

The ideal deployment context is an offline or asynchronous evaluation pipeline, not a real-time user-facing code path. You should wire this prompt into a batch evaluation harness that samples production queries, retrieves evidence from your current pipeline, and runs the scorecard against a representative query set. Store the structured scores alongside query metadata, retriever configuration, and timestamp so you can track retrieval quality trends over time, compare retriever versions, and set objective thresholds for promotion to production.

Do not use this prompt for real-time answer generation or for making binary answer/don't-answer decisions. It is a diagnostic instrument, not a gating mechanism. Pair it with a downstream gating prompt—such as the RAG Sufficiency Gate or Answer Abstention Decision Prompt—when you need a production guardrail. Also avoid using this prompt on queries where the retrieval set is trivially empty; the scorecard assumes at least one candidate passage exists to evaluate. For empty retrieval sets, route directly to a missing-context or refusal workflow.

This template is designed to be dropped directly into your RAG evaluation pipeline or monitoring harness. It instructs the model to act as an evidence auditor, evaluating a set of retrieved passages against a user query across five calibrated dimensions. The output is a structured JSON scorecard suitable for dashboards, continuous monitoring, and SLA tracking. Replace each square-bracket placeholder with your actual data before sending the prompt to the model.

textCopy
You are an evidence sufficiency auditor. Your task is to evaluate whether the provided evidence passages are sufficient to answer the user query. Produce a structured scorecard assessing sufficiency across five dimensions.

## INPUT

**User Query:**
[USER_QUERY]

**Retrieved Evidence Passages:**
[EVIDENCE_PASSAGES]

**Domain Context (optional):**
[DOMAIN_CONTEXT]

## OUTPUT SCHEMA

Return a valid JSON object with this exact structure:
{
  "overall_sufficiency_score": <float 0.0-1.0>,
  "overall_verdict": "sufficient" | "partially_sufficient" | "insufficient",
  "dimensions": {
    "relevance": {
      "score": <float 0.0-1.0>,
      "rationale": "<explanation of how well the evidence addresses the query topic and intent>"
    },
    "completeness": {
      "score": <float 0.0-1.0>,
      "rationale": "<explanation of whether all aspects of the query are covered by the evidence>"
    },
    "authority": {
      "score": <float 0.0-1.0>,
      "rationale": "<explanation of source credibility, expertise indicators, and trustworthiness>"
    },
    "recency": {
      "score": <float 0.0-1.0>,
      "rationale": "<explanation of whether the evidence is current enough for the query's temporal requirements>"
    },
    "specificity": {
      "score": <float 0.0-1.0>,
      "rationale": "<explanation of how precisely the evidence addresses the query versus providing general or tangential information>"
    }
  },
  "identified_gaps": [
    "<specific missing information needed to fully answer the query>"
  ],
  "recommendation": "<actionable guidance: proceed to answer, request more evidence, or refuse>"
}

## CONSTRAINTS

- Score each dimension independently. Do not let one dimension bias another.
- For recency: if the query is time-sensitive, penalize outdated evidence. If the query is timeless, score recency high by default and note this in the rationale.
- For authority: consider source reputation, author expertise, publication venue, and presence of citations. Flag sources with no identifiable authority.
- For completeness: identify every sub-question or information need in the query and check whether evidence addresses each one.
- If evidence is missing critical facts, entities, or temporal context, list these explicitly in identified_gaps.
- overall_sufficiency_score must reflect the weighted combination of dimension scores, not an independent judgment.
- If the evidence is insufficient, the recommendation must explain what additional evidence is needed.
- Do not answer the user query. Only assess evidence sufficiency.

## RISK LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high": apply stricter thresholds. Score authority and completeness more conservatively. Flag any unsupported inference risk. Recommend human review when overall_sufficiency_score is below 0.8.

Adaptation guidance: Replace [USER_QUERY] with the end user's question or search intent. [EVIDENCE_PASSAGES] should contain the full text of retrieved chunks, each prefixed with a source identifier for traceability. [DOMAIN_CONTEXT] is optional but valuable for specialized domains such as legal, medical, or financial where authority and specificity standards differ. [RISK_LEVEL] accepts low, medium, or high and adjusts the scoring conservatism accordingly. For production use, wrap this prompt in a validation layer that checks the output JSON against the schema before the scorecard enters your monitoring pipeline. If the model returns malformed JSON, use a repair prompt or retry with stricter format instructions before surfacing results.

The Evidence Sufficiency Scorecard Prompt is designed to operate as a post-retrieval, pre-generation quality gate in a RAG pipeline. After your retriever returns a set of candidate passages but before those passages are sent to an answer-generation model, this prompt evaluates whether the evidence is sufficient. The output is a structured scorecard that can be used for automated routing decisions (answer, request more context, or refuse), dashboard population for retrieval quality monitoring, and SLA tracking. The prompt expects a query, a list of retrieved passages with metadata, and an optional output schema to enforce the scorecard structure.

Implementation flow: 1) Receive user query and retrieve top-k passages. 2) Assemble the prompt by injecting [QUERY], [RETRIEVED_PASSAGES] (with source, date, and authority metadata), and [OUTPUT_SCHEMA] (a JSON schema defining the scorecard fields). 3) Call a capable model (GPT-4o, Claude 3.5 Sonnet, or equivalent) with response_format set to json_object and temperature at or below 0.2 for consistency. 4) Validate the output against your scorecard schema—check that all required dimensions (relevance, completeness, authority, recency, specificity) are present, scores are within expected ranges, and the overall sufficiency verdict is present. 5) If validation fails, retry once with the validation error message appended to the prompt as a correction instruction. 6) Log the full scorecard, retrieval set metadata, and model call details to your observability platform for trace analysis and regression testing. 7) Route based on the overall sufficiency verdict: if sufficient, proceed to answer generation; if insufficient, trigger a gap report or clarification flow.

Critical guardrails: This prompt is a structured evaluation step, not a user-facing response. Do not expose raw scorecard output to end users without interpretation. For high-stakes domains (healthcare, legal, finance), always include a human-review step when the scorecard shows borderline sufficiency or when authority scores are low. Implement a circuit breaker that escalates to human review if the model returns malformed JSON after two retries. Use the scorecard dimensions as monitoring metrics over time: track average completeness scores per knowledge base, detect drift when authority scores decline, and set alerts when the rate of insufficient verdicts spikes. Finally, pair this prompt with the Evidence Sufficiency Rubric for LLM Judge to run automated evals on the scorecard quality itself—ensuring the scorecard prompt remains calibrated as your retrieval pipeline evolves.