Evidence Sufficiency Rubric for LLM Judge

This prompt is designed for evaluation engineers and AI quality teams who need to build automated, reproducible RAG quality pipelines. The core job-to-be-done is calibrating an LLM-as-judge to assess whether a set of retrieved passages is sufficient to answer a user's query. Instead of relying on spot-checking or simple binary relevance, this rubric forces the judge to evaluate evidence across multiple dimensions—factual coverage, specificity, authority, and recency—and produce a structured score with detailed reasoning. The ideal user is someone running batch evaluations against a golden dataset, monitoring retrieval quality in production, or setting up a gating mechanism that prevents ungrounded answers from reaching end users.

You should use this prompt when you need a consistent, multi-dimensional standard for evidence quality that goes beyond a single relevance score. It is appropriate for offline evaluation pipelines where you are comparing retrieval system variants, tuning chunking strategies, or benchmarking embedding models. It is also suitable as a pre-generation guardrail in production, where the rubric's output can route queries to fallback, human review, or clarification flows. The prompt expects structured inputs: a user query, a set of retrieved passages with metadata, and a defined output schema. The rubric's scoring anchors—from 'fully sufficient' to 'completely insufficient'—are designed to reduce judge inconsistency across runs.

Do not use this prompt as a real-time, user-facing response generator. It is an evaluation and gating tool, not a conversational answer prompt. Avoid it when you need a simple binary yes/no on answerability without dimensional detail; a lighter classifier prompt is more appropriate for low-latency pre-flight checks. This prompt also assumes the retrieved passages are already filtered for basic relevance—it assesses sufficiency, not initial retrieval quality. If your retrieval pipeline has not removed obviously off-topic passages, apply a relevance filter first. Finally, in high-stakes domains like healthcare or legal, the rubric's output should inform a human review decision, not replace it. The next step after reading this section is to examine the prompt template and adapt the scoring dimensions to your specific domain's evidence standards.

This prompt template implements a calibrated evidence sufficiency rubric designed for automated RAG evaluation pipelines. It instructs an LLM judge to score a retrieval set against a query across multiple completeness dimensions, producing a structured scorecard with dimension-level ratings, an overall sufficiency determination, and a detailed rationale. The rubric anchors are designed to reduce judge leniency bias and improve inter-rater reliability when used at scale. Use this template as the core evaluation instruction inside your LLM-as-judge harness, replacing the square-bracket placeholders with your specific query, retrieved passages, and evaluation parameters before each run.

textCopy
You are an evaluation judge assessing whether retrieved evidence is sufficient to answer a user query. Your task is to produce a structured, calibrated sufficiency scorecard. Do not answer the query. Only evaluate the evidence.

## INPUT

**Query:** [QUERY]

**Retrieved Evidence Passages:**
[RETRIEVED_PASSAGES]

## SCORING DIMENSIONS

For each dimension, assign a score from 1 (critically insufficient) to 5 (fully sufficient). Provide a brief justification for each score.

1. **Factual Coverage:** Does the evidence contain the specific facts, entities, numbers, or claims needed to address the query? Score 1 if key facts are entirely missing. Score 5 if all required facts are present with supporting detail.

2. **Specificity Alignment:** Does the evidence match the specificity level of the query? Score 1 if the evidence is too general or off-topic. Score 5 if the evidence directly addresses the query at the right granularity.

3. **Temporal Relevance:** Is the evidence current enough for the query's time sensitivity? Score 1 if the evidence is clearly outdated for a time-sensitive query. Score 5 if the evidence is recent or the query has no time dependency. Note "N/A" if time is irrelevant.

4. **Authority and Trustworthiness:** Does the evidence come from sources appropriate for the query domain? Score 1 if sources are clearly unreliable or inappropriate. Score 5 if sources are authoritative and credible for the domain.

5. **Absence of Critical Gaps:** Are there any missing facts, entities, or perspectives that would prevent a complete and accurate answer? Score 1 if critical gaps make answering impossible. Score 5 if no significant gaps exist.

## OVERALL SUFFICIENCY DETERMINATION

Based on the dimension scores, classify the evidence into one of these categories:
- **SUFFICIENT:** All key information is present. A grounded answer can be produced without speculation.
- **PARTIALLY SUFFICIENT:** Some information is present, but gaps require caveats, partial answers, or expressed uncertainty.
- **INSUFFICIENT:** Critical gaps prevent any grounded answer. The system should refuse or request clarification.

## OUTPUT FORMAT

Return a valid JSON object with this exact schema:

{
  "overall_sufficiency": "SUFFICIENT" | "PARTIALLY SUFFICIENT" | "INSUFFICIENT",
  "overall_rationale": "Concise explanation of the overall determination.",
  "dimension_scores": {
    "factual_coverage": { "score": 1-5, "justification": "..." },
    "specificity_alignment": { "score": 1-5, "justification": "..." },
    "temporal_relevance": { "score": 1-5 | "N/A", "justification": "..." },
    "authority_trustworthiness": { "score": 1-5, "justification": "..." },
    "absence_of_critical_gaps": { "score": 1-5, "justification": "..." }
  },
  "missing_information": ["List specific missing facts, entities, or context"],
  "confidence": 0.0-1.0
}

## CONSTRAINTS

- Do not answer the user query. Only evaluate the evidence.
- If the query is ambiguous, note this in the rationale but evaluate the evidence as provided.
- If evidence passages contradict each other, flag this in missing_information and adjust scores accordingly.
- The confidence score should reflect your certainty in the overall sufficiency determination, not answer correctness.
- Be strict: leniency produces false confidence in production systems.

After copying this template, adapt it by replacing [QUERY] with the user's original question and [RETRIEVED_PASSAGES] with the top-k passages from your retrieval pipeline, formatted with source identifiers. For production evaluation pipelines, wrap this prompt in a harness that validates the output JSON against the schema, retries on parse failures, and logs dimension scores to your monitoring dashboard. If your domain requires additional dimensions—such as regulatory compliance or safety alignment—add them to the dimension_scores object while preserving the 1–5 scale for calibration consistency. For high-stakes applications, run this judge against a human-annotated golden set to measure alignment before relying on automated scores.

The Evidence Sufficiency Rubric prompt is designed to run as a post-retrieval, pre-generation evaluation step inside an automated RAG quality pipeline. It should be called after the retrieval system returns a set of passages but before the answer generation model consumes them. The rubric prompt receives the user query and the retrieved evidence set, then returns a structured scorecard with dimension-level ratings and an overall sufficiency classification. This output becomes a gating signal: if sufficiency is below a configured threshold, the pipeline can trigger re-retrieval with expanded queries, route to a human review queue, or return a calibrated refusal response instead of generating an ungrounded answer. The prompt is not intended for real-time user-facing latency budgets under 500ms; it is designed for offline evaluation, batch scoring of retrieval quality, and pre-release regression testing where thoroughness matters more than speed.

To integrate this prompt into an application, wrap it in a validation and retry harness. The expected output is a JSON object with fields for each rubric dimension (factual coverage, specificity, temporal relevance, authority, and contradiction presence), an overall sufficiency score on a 1–5 scale, and a binary sufficient flag. Implement a JSON schema validator that checks for missing fields, out-of-range scores, and malformed rationales. If validation fails, retry once with the error message appended to the prompt as a correction hint. Log every rubric output alongside the retrieval set ID, query, and model version for traceability. For high-stakes domains such as healthcare or legal review, add a human-in-the-loop step: when the rubric returns a borderline score (e.g., 3 out of 5) or flags contradictions, route the case to a review queue with the rubric output and source passages attached. Model choice matters here—use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent) because weaker models tend to drift toward lenient scores or omit dimension-level breakdowns. Avoid using this rubric as the sole quality signal; pair it with answer-level faithfulness verification and periodic human calibration samples to detect rubric drift over time.

Before deploying this rubric prompt to production, build a calibration eval set of 50–100 query-evidence pairs with human-annotated sufficiency scores and dimension ratings. Run the rubric against this set and measure agreement using quadratic weighted kappa for the 1–5 scale and F1 for the binary sufficient flag. If agreement drops below 0.7 kappa, adjust the rubric anchors or add few-shot examples of borderline cases. Monitor the rubric's score distribution in production—shifts toward extreme scores (all 1s or all 5s) often indicate the model is ignoring the evidence and applying a heuristic. The most common failure mode is leniency bias, where the model rates insufficient evidence as adequate because the passages mention the topic without actually containing the answer. Mitigate this by including negative examples in the prompt and by periodically injecting known-insufficient retrieval sets into the eval pipeline as canaries.