Source Grounding Score Prompt for LLM Outputs

This prompt is designed for evaluation harnesses and CI/CD pipelines that need a numeric, reproducible score for how well every sentence in a generated text is anchored to provided source documents. The primary job-to-be-done is automated quality gating: before an LLM-generated answer, summary, or report reaches a user or a downstream system, you run this prompt to verify that claims are not hallucinated. The ideal user is an AI engineer or ML ops lead building a RAG system, a document Q&A product, or a compliance-sensitive text-generation pipeline where unsupported statements carry regulatory, financial, or reputational risk. You need the full set of source documents that were available to the generation model, the generated text itself, and a defined grounding rubric.

Do not use this prompt when you lack the original source documents or when the generated text is creative, speculative, or opinion-based by design. This prompt measures factual grounding against provided evidence; it cannot assess whether the sources themselves are correct, complete, or authoritative. It is also a poor fit for real-time chat moderation where latency constraints preclude a full evaluation pass. For high-risk domains such as healthcare, legal, or finance, treat the grounding score as a necessary but not sufficient check—always pair it with human review for outputs that fall below your calibrated threshold. The prompt works best as a gate in a batch evaluation pipeline, not as a synchronous user-facing validator.

What you need before using this prompt: (1) The generated text to evaluate, split into individual sentences or atomic claims. (2) The complete set of source documents or chunks that were provided to the generation model. (3) A grounding rubric defining what counts as 'fully grounded,' 'partially grounded,' and 'not grounded.' (4) A calibration set of human-annotated grounding judgments to validate that the prompt's scores align with your risk tolerance. Next step: After reading this playbook, copy the prompt template, adapt the rubric to your domain, run it against a calibration set of 50-100 examples, and tune the per-sentence and aggregate thresholds before wiring it into your production evaluation harness.

This prompt template is the core of the Source Grounding Score evaluation harness. It instructs the model to act as a strict evaluator, comparing a generated text against a set of source documents sentence by sentence. The output is a structured JSON object containing per-sentence grounding scores, a rationale for each score, and an aggregate document-level score. The template uses square-bracket placeholders for all dynamic inputs, making it ready to be wired into an automated evaluation pipeline where the generated text and its source context are injected programmatically.

textCopy
You are a strict evaluator of factual grounding. Your task is to compare a generated text against a set of provided source documents and assign a grounding score to every sentence.

## INPUT
**Generated Text:**
[GENERATED_TEXT]

**Source Documents:**
[SOURCE_DOCUMENTS]

## SCORING RUBRIC
For each sentence in the Generated Text, assign a score from 0 to 1 based on how well it is supported by the Source Documents:
- **1.0 (Fully Grounded):** All factual claims in the sentence are directly and unambiguously supported by the sources.
- **0.5 (Partially Grounded):** Some claims are supported, but others are missing, or the support is implicit and requires significant inference.
- **0.0 (Not Grounded):** The sentence contains factual claims with no support in the sources, or it directly contradicts them.

## OUTPUT_SCHEMA
Return a single JSON object with the following structure:
{
  "aggregate_score": <float between 0.0 and 1.0>,
  "sentence_scores": [
    {
      "sentence_index": <integer>,
      "sentence_text": "<exact text of the sentence>",
      "score": <float 0.0, 0.5, or 1.0>,
      "rationale": "<brief explanation of the score, citing the specific source passage or noting its absence>"
    }
  ]
}

## CONSTRAINTS
- Do not modify the Generated Text. Score it exactly as provided.
- The `aggregate_score` must be the mean of all `sentence_scores`.
- If a sentence is purely stylistic, transitional, or a header without factual claims, assign a score of 1.0 and note this in the rationale.
- If the Generated Text is empty, return an empty `sentence_scores` array and an `aggregate_score` of 1.0.

To adapt this template, replace the [GENERATED_TEXT] and [SOURCE_DOCUMENTS] placeholders with your actual data. The [SOURCE_DOCUMENTS] should be a single string containing all retrieved context, clearly delimited by document identifiers. For high-stakes workflows, you can extend the [OUTPUT_SCHEMA] to include a [CONFIDENCE] field for each sentence score, allowing a secondary evaluator or human reviewer to focus on low-confidence judgments. The rubric can also be adapted for domain-specific needs, such as penalizing medical or legal claims that lack a direct quote from the source, by adjusting the score definitions and adding a [RISK_LEVEL] parameter that triggers stricter evaluation criteria.

The Source Grounding Score prompt is designed to operate as a post-generation evaluation step, not as part of the primary generation request. In a production RAG or document Q&A system, you should call this prompt after the main answer is generated and its citations are resolved. The typical wiring pattern is: (1) the system generates an answer with citations, (2) a post-processing step extracts the answer text and the list of source chunks that were provided to the generation model, (3) this prompt is called with the answer and sources as inputs, and (4) the resulting per-sentence and aggregate grounding scores are logged alongside the interaction for monitoring, gating, or human review routing.

Validation and gating are critical because a grounding score alone does not block bad outputs. Implement a threshold gate after scoring: if the aggregate grounding score falls below your defined minimum (e.g., 0.7 for high-risk domains, 0.5 for exploratory use), route the output for human review or suppress it entirely. For per-sentence scores, flag any sentence scoring below 0.3 as a likely hallucination and either remove it or append an [UNVERIFIED] marker before showing it to users. Store the raw scoring output—including the model's reasoning for each sentence—in your observability system so you can debug false positives and false negatives later. Model choice matters: use a capable instruction-following model (GPT-4, Claude 3.5 Sonnet, or equivalent) for the scoring step, even if your generation model is smaller, because scoring requires careful reasoning about entailment and contradiction.

Retry logic should be minimal for this prompt. If the output fails to parse as valid JSON with the expected schema, retry once with a stronger format instruction. If it still fails, log the raw output and escalate—do not silently fall back to a default score. Calibration is an ongoing concern: periodically run a golden set of human-annotated sentence-source pairs through this prompt and compare the model's grounding scores against human judgments. Track drift over time, especially after model version updates. When grounding scores degrade, investigate whether the issue is in the generation step (producing unsupported claims), the retrieval step (returning irrelevant sources), or the scoring step itself (model behavior change). Wire these calibration results into your prompt regression test suite so that a drop in scoring accuracy triggers an alert before it affects users.