Source Support Scoring Prompt

This prompt is designed for evaluation engineers and RAG quality teams who need to move beyond binary 'supported/unsupported' flags and into calibrated, numerical measurement of evidence quality. The core job-to-be-done is assigning a support score to each claim-source pair, measuring how directly a cited passage substantiates a specific statement. This is an offline evaluation tool, not a real-time filtering component. You should use it when building automated evaluation pipelines, conducting inter-annotator agreement studies to calibrate human reviewers against model judgments, or performing regression testing to detect when a change to your retrieval or generation stack silently degrades evidence quality. The ideal user is someone who owns a ground-truth dataset of claim-source pairs and needs repeatable, structured scoring to monitor system health over time.

The prompt requires a specific set of inputs to function reliably. You must provide the claim to be evaluated, the full text of the cited source passage, and a detailed scoring rubric that defines what each numerical score means in terms of evidence strength. Without a clear rubric, the model will default to inconsistent, uncalibrated judgments that cannot be compared across runs. The output is structured JSON containing the numerical score, a concise justification tied to specific rubric criteria, and optional fields for confidence and evidence span references. This structured output is designed to be consumed directly by evaluation scripts, inserted into databases for trend analysis, or used as input to an inter-annotator agreement calculator. Do not use this prompt for end-user-facing answer generation, real-time retrieval filtering where latency is measured in milliseconds, or any workflow where the model's reasoning is shown directly to non-technical users without review.

The primary constraint is that this prompt measures support, not truth. A passage can receive a high support score for a claim that is factually incorrect if the passage genuinely and directly supports it. This prompt does not perform fact-checking against external knowledge; it only evaluates the relationship between the provided claim text and the provided source text. If you need to verify factual accuracy, you must pair this prompt with a separate fact-checking or claims verification workflow. Additionally, this prompt is sensitive to rubric design. A vague rubric like 'score 1-5 for support' will produce noisy, unusable results. Invest time in defining each score level with concrete criteria, such as 'Score 4: The passage explicitly states the claim using substantially similar language' versus 'Score 2: The passage mentions a related concept but requires inference to connect to the claim.' Start by running the prompt against a small, human-annotated calibration set, measure agreement, and iterate on your rubric definitions before scaling to your full dataset.

This prompt template is designed to be dropped directly into an evaluation harness for source support scoring. It accepts a claim and a source passage, then returns a numerical support score along with a structured justification. The template uses square-bracket placeholders that you replace with your data before each evaluation run. The rubric is calibrated for inter-annotator agreement, making it suitable for both human review and LLM-as-judge pipelines.

textCopy
You are an evidence evaluation specialist. Your task is to score how directly a cited source passage supports a specific claim.

## INPUT
Claim: [CLAIM_TEXT]
Source Passage: [SOURCE_PASSAGE_TEXT]
Source Metadata: [SOURCE_METADATA]

## SCORING RUBRIC
Score the claim-source pair on a 1-5 scale:
- 5 (Directly Supports): The passage explicitly states the claim or provides direct evidence that proves it.
- 4 (Strongly Implies): The passage does not state the claim verbatim but contains information from which the claim follows logically with minimal inference.
- 3 (Partially Supports): The passage provides some relevant evidence but requires significant inference, or only supports part of the claim.
- 2 (Weakly Related): The passage is on the same topic but does not provide meaningful evidence for or against the claim.
- 1 (No Support / Contradicts): The passage either provides no relevant information or actively contradicts the claim.

## CONSTRAINTS
- Score based ONLY on the provided passage, not external knowledge.
- If the passage is insufficient to evaluate the claim, score 1 and explain why.
- Do not infer information not present in the passage.
- Consider both explicit statements and necessary logical implications.

## OUTPUT_SCHEMA
Return valid JSON with this exact structure:
{
  "score": <integer 1-5>,
  "support_level": "<Directly Supports|Strongly Implies|Partially Supports|Weakly Related|No Support / Contradicts>",
  "rationale": "<2-4 sentence explanation citing specific passage content>",
  "key_evidence": "<direct quote or paraphrase from the passage that most supports this score>",
  "missing_information": "<what the passage would need to contain for a higher score, or null if score is 5>",
  "confidence": <float 0.0-1.0 indicating scorer confidence in this assessment>
}

## EXAMPLES
Example 1:
Claim: "The company reported revenue of $4.2 billion in Q3 2024."
Source Passage: "Acme Corp announced third-quarter revenue of $4.2 billion, up 12% year-over-year, driven by strong cloud services growth."
Output:
{
  "score": 5,
  "support_level": "Directly Supports",
  "rationale": "The passage explicitly states 'third-quarter revenue of $4.2 billion,' which directly matches the claim. No inference is required.",
  "key_evidence": "Acme Corp announced third-quarter revenue of $4.2 billion",
  "missing_information": null,
  "confidence": 0.98
}

Example 2:
Claim: "The company's cloud division is its fastest-growing segment."
Source Passage: "Cloud services revenue grew 12% year-over-year, while hardware revenue grew 3% and consulting declined 2%."
Output:
{
  "score": 4,
  "support_level": "Strongly Implies",
  "rationale": "The passage shows cloud services growing at 12%, which is higher than hardware (3%) and consulting (-2%). While it doesn't explicitly state cloud is the fastest-growing, the comparative growth rates strongly imply this conclusion.",
  "key_evidence": "Cloud services revenue grew 12% year-over-year, while hardware revenue grew 3% and consulting declined 2%",
  "missing_information": "Explicit statement that cloud is the fastest-growing segment",
  "confidence": 0.85
}

Example 3:
Claim: "The CEO announced plans to acquire a competitor."
Source Passage: "In the earnings call, the CEO discussed expanding market share through organic growth and strategic partnerships."
Output:
{
  "score": 1,
  "support_level": "No Support / Contradicts",
  "rationale": "The passage discusses organic growth and partnerships, which are alternatives to acquisition. It does not mention any acquisition plans and the emphasis on organic growth may weakly contradict the claim.",
  "key_evidence": "expanding market share through organic growth and strategic partnerships",
  "missing_information": "Any mention of acquisition plans, target companies, or M&A strategy",
  "confidence": 0.92
}

To adapt this template, replace the placeholders with your evaluation data. [CLAIM_TEXT] should contain a single factual claim extracted from a model output. [SOURCE_PASSAGE_TEXT] should contain the full text of the cited passage. [SOURCE_METADATA] is optional but useful for tracking source authority, date, and document context. For batch evaluation, wrap this prompt in a loop that processes claim-source pairs and aggregates scores. If you are using this for inter-annotator agreement, run the same pairs through multiple evaluators and compute Cohen's kappa or Krippendorff's alpha on the score distribution.

Before deploying this prompt in production, validate that the output JSON parses correctly and that scores align with human-annotated ground truth on at least 50 claim-source pairs. Watch for common failure modes: the model may score based on its own knowledge rather than the provided passage, or it may inflate scores for partially relevant passages. Add a post-processing check that rejects outputs where the rationale references information not present in the source passage. For high-stakes domains such as legal or medical verification, always include human review of low-confidence scores and any score of 3 or below.

The Source Support Scoring Prompt is not a user-facing prompt; it is an evaluation instrument designed to run inside an automated quality monitoring pipeline. Its primary job is to assign a numerical support score to each claim-source pair, measuring how directly a cited passage supports a specific statement. In production, this prompt typically runs after a RAG system has generated an answer with citations, and its output feeds into a grounding verification dashboard, a regression test suite, or a pre-release quality gate. The harness must treat each scoring call as a measurement that requires validation, not as a conversational completion.

Wire the prompt into your evaluation pipeline by wrapping it in a function that accepts a list of claim-source pairs and returns a structured scoring record. Each call should include: the claim text, the source passage text, the scoring rubric (with anchor definitions for each score level), and the expected output schema. Implement a JSON schema validator on the response to enforce the required fields: claim_id, source_id, support_score, rationale, and key_evidence_span. If the model returns a score outside the defined range or omits the rationale, reject the response and retry with a stricter schema reminder. Log every raw response, validation failure, and retry attempt for later failure mode analysis. For high-stakes domains such as healthcare or legal review, route all scores below a configurable threshold to a human review queue before they are accepted into the evaluation dataset.

Calibration is the hardest part of this harness. Before trusting automated scores, run the prompt against a golden dataset of 50-100 claim-source pairs that have been independently scored by at least two human annotators. Calculate Cohen's kappa or Krippendorff's alpha between the model's scores and the human consensus. If agreement falls below 0.7, adjust the rubric anchors, add few-shot examples of borderline cases, or tighten the definition of each score level. Recalibrate after any prompt change, model upgrade, or retrieval pipeline modification. Store calibration results alongside prompt version hashes so you can detect drift over time. The harness should also compute inter-annotator agreement across multiple model runs with different temperature settings to quantify score stability.

Integrate the scoring output into your broader evaluation system by aggregating scores per claim, per source, and per answer. A single unsupported claim in an otherwise well-grounded answer is a red flag that should trigger a deeper faithfulness check. Use the key_evidence_span field to power a citation precision metric: if the model identifies a supporting span but that span does not actually contain the claim's factual content, the citation is misleading even if the score is high. Build a dashboard that tracks support score distributions over time, broken down by retrieval strategy, document source, and model version. When scores degrade, the harness should alert the team and surface the lowest-scoring claim-source pairs for manual review. Avoid the temptation to treat the support score as a single pass/fail gate; it is a diagnostic signal that works best when combined with hallucination detection, citation verification, and human spot-checking in a layered evaluation architecture.