LLM-as-Judge Evidence Scoring Prompt Template

This prompt is a measurement instrument for evaluation engineers and RAG pipeline architects who need to automate evidence quality assessment at scale. Use it when you must rank retrieved passages by their support for a specific claim, calibrate those scores against human judgments, and detect systematic biases like position or length effects. The core job-to-be-done is not generating an answer for an end user, but producing a structured, explainable, and auditable quality signal that can be fed into an offline evaluation suite, a CI/CD quality gate, or a periodic production audit. You should reach for this prompt when a simple relevance classifier or an embedding cosine similarity score is insufficient—specifically, when you need multi-factor scores, pairwise comparisons, confidence intervals, or a detailed rubric-based justification that your team can review and debug.

Deploy this prompt in controlled, non-real-time evaluation pipelines first. A typical integration point is an offline scoring job that runs nightly over a sample of production retrieval sets, comparing the LLM's evidence scores against a human-annotated golden dataset. This allows you to calibrate the prompt's behavior, tune its rubrics, and measure inter-rater reliability before it ever influences a live user response. In a CI/CD context, you can gate a RAG pipeline deployment by running this prompt against a fixed set of claim-evidence pairs and failing the check if the weighted F1 score or Spearman rank correlation drops below a predefined threshold. For production audits, use it as a sampling profiler—pull 100 random retrieval events, score them, and generate a report on evidence quality drift. In all cases, the prompt's output should be logged as structured JSON, not free text, so that score distributions, confidence intervals, and bias metrics can be trended over time in your observability stack.

Do not use this prompt when a lightweight relevance signal is sufficient for your use case. If you only need to filter out obviously irrelevant passages before answer generation, a smaller, faster cross-encoder model or a thresholded vector similarity score will be cheaper and lower latency. The overhead of an LLM-as-Judge—token costs, inference time, and the complexity of parsing structured output—is warranted only when you need the explainability, multi-factor reasoning, and calibrated confidence that a general-purpose model provides. Similarly, avoid this prompt in synchronous, user-facing request paths where latency budgets are under 500 milliseconds; the scoring and comparison logic is designed for asynchronous evaluation, not real-time re-ranking. Finally, if your evidence set is small and static, and you can afford a one-time human annotation pass, do that instead. This prompt's value compounds when you have a large, dynamic retrieval corpus and need automated, repeatable quality measurement that scales beyond what a human review team can handle.

This template instructs the model to act as a calibrated judge, evaluating how well each evidence passage supports a given claim. It uses a defined scoring rubric to produce consistent, structured outputs that can be directly consumed by downstream ranking, filtering, or audit systems. The template is designed to be wired into an automated evaluation pipeline where evidence quality must be assessed programmatically, not just read by a human.

textCopy
You are a calibrated evidence scoring judge. Your task is to evaluate how well each provided evidence passage supports a specific claim.

## INPUT
Claim: [CLAIM]
Evidence Passages:
[EVIDENCE_PASSAGES]

## SCORING RUBRIC
Score each passage on a scale of 1-5 using these criteria:
- 5 (Directly Supports): The passage explicitly confirms the claim with specific, matching details.
- 4 (Strongly Consistent): The passage is fully consistent with the claim and provides substantial supporting context, but lacks explicit confirmation.
- 3 (Partially Relevant): The passage touches on the claim's topic but neither clearly supports nor contradicts it.
- 2 (Weakly Inconsistent): The passage contains information that mildly contradicts or undermines the claim.
- 1 (Directly Contradicts): The passage explicitly refutes the claim with specific, conflicting details.
- 0 (Irrelevant): The passage has no meaningful relationship to the claim.

## OUTPUT SCHEMA
Return a valid JSON object with this exact structure:
{
  "scores": [
    {
      "passage_index": 0,
      "score": 5,
      "confidence": [0.85, 0.95],
      "rationale": "One-sentence explanation of the score.",
      "key_quote": "The most relevant sentence from the passage."
    }
  ],
  "summary": "Brief overall assessment of evidence strength for the claim."
}

## CONSTRAINTS
- Score every passage independently.
- Confidence must be a two-element array: [lower_bound, upper_bound] between 0.0 and 1.0.
- If you are uncertain, widen the confidence interval rather than guessing.
- The key_quote must be a verbatim excerpt from the passage.
- Do not include any text outside the JSON object.

To adapt this template, replace the [CLAIM] and [EVIDENCE_PASSAGES] placeholders with your actual inputs. For production use, add a [FEW_SHOT_EXAMPLES] section before the output schema to calibrate the model against your specific domain's evidence patterns. If scoring more than 5-7 passages, batch them to avoid position bias and length effects that degrade scoring consistency in later passages. Always validate the output JSON against the schema before accepting scores, and implement a retry with a stricter constraint prompt if the model deviates from the required format.

The LLM-as-Judge Evidence Scoring prompt is not a one-shot call. It belongs inside a structured evaluation harness that validates outputs, calibrates scores against human judgments, and detects systematic biases before scores feed downstream ranking or filtering systems. The harness should treat each prompt invocation as a single scoring event with traceable inputs, outputs, and metadata. Store the raw prompt, the model response, the parsed scores, and any validation errors in an evaluation log. This log becomes your audit trail for debugging score drift, model regressions, and bias patterns over time.

Start by wrapping the prompt call in a function that accepts a query, a list of evidence passages, and a scoring rubric. The function should: (1) assemble the prompt with the template's square-bracket placeholders filled, (2) call the model with a low temperature (0.0–0.2) for scoring consistency, (3) parse the structured output into a typed object with per-passage scores, confidence intervals, and justification fields, and (4) run a schema validator against the parsed output. If the output fails schema validation, retry once with the validation error appended to the prompt as a correction instruction. If the retry also fails, log the failure and flag the passage batch for human review rather than silently accepting malformed scores. For high-stakes domains—legal, medical, financial—require human sign-off on score distributions before they influence answer generation or retrieval re-ranking.

Calibration against human judgments is the most important harness component. Maintain a golden dataset of query-passage pairs with human-assigned scores. After each prompt version change or model upgrade, run the scoring prompt against this dataset and compute agreement metrics: Cohen's kappa for categorical tiers, mean absolute error for numerical scores, and position-bias checks that compare scores when the same passage appears first versus last in the input list. If agreement drops below a configured threshold, block the prompt version from production and investigate. Common failure modes include length bias where longer passages receive inflated scores, position bias where earlier passages score higher, and confidence miscalibration where the model reports high confidence on clearly ambiguous evidence. Your harness should explicitly test for these biases on every evaluation run, not just during initial development.