Claim Extraction and Source Verification Prompt

This prompt is designed for eval pipeline builders and QA engineers who need an automated, repeatable method to audit AI-generated answers for factual fidelity. The core job-to-be-done is decomposing a generated text into discrete, verifiable claims and then matching each claim against a provided set of source evidence. The output is a structured audit trail—a claim list with per-claim grounding scores—that can be used as a quality gate in a CI/CD pipeline or as a continuous monitoring signal in production. The ideal user is someone integrating this step into an automated testing harness, not a human performing a one-off manual review.

Use this prompt when you have a generated answer and the exact source context that was used to produce it, and you need a machine-readable verdict on whether each factual statement is supported. This is a post-generation verification step, distinct from real-time RAG answer synthesis. It is appropriate for regression testing where you compare model outputs against a golden dataset of known-good answers and sources, for pre-release answer quality gates that block deployment if hallucination rates exceed a threshold, and for production observability where you sample live answers and run them through this audit loop. Do not use this prompt when you lack the original source evidence, when the answer contains subjective opinions rather than verifiable claims, or when you need a single overall score without per-claim traceability. It is also not a substitute for human review in regulated domains—it is a filtering and prioritization tool that should escalate uncertain cases to human auditors.

Before wiring this into your application, define your grounding score thresholds and escalation rules. A claim with a low grounding score should trigger a specific action: block the answer, flag for human review, or regenerate with stricter constraints. The prompt template includes placeholders for [INPUT_ANSWER], [SOURCE_CONTEXT], and [OUTPUT_SCHEMA] so you can adapt it to your specific document formats and required JSON fields. Start by running this against a labeled evaluation set where you already know which claims are supported and which are fabricated, then calibrate your thresholds before relying on it in a production gate.

This template is the core instruction set for decomposing an AI-generated answer into discrete, verifiable claims and then checking each claim against a provided set of source documents. It is designed to be dropped into an evaluation pipeline, not a user-facing chat interface. The prompt enforces a strict JSON output schema, making its results machine-readable for automated quality gates. Use this when you need a programmatic, repeatable method for factual auditing before an answer reaches a user or a downstream system.

codeCopy
You are an expert fact-checker and claims auditor. Your task is to perform a structured audit of a generated answer against a set of provided source documents. You will decompose the answer into a list of discrete, atomic claims and then verify each claim against the evidence.

# INPUT

## Generated Answer to Audit
[ANSWER]

## Source Evidence
[SOURCE_DOCUMENTS]

# OUTPUT SCHEMA

Return a single JSON object with the following structure. Do not include any text outside the JSON object.

{
  "claims": [
    {
      "claim_id": "string (unique identifier, e.g., 'C1', 'C2')",
      "claim_text": "string (the exact, self-contained factual statement from the answer)",
      "verification_status": "string (one of: 'SUPPORTED', 'UNSUPPORTED', 'CONTRADICTED', 'NOT_ADDRESSED')",
      "source_evidence": [
        {
          "source_id": "string (the identifier of the source document from [SOURCE_DOCUMENTS])",
          "relevant_passage": "string (the exact text from the source that supports or contradicts the claim)",
          "match_explanation": "string (a brief explanation of how the passage relates to the claim)"
        }
      ],
      "confidence_score": "float (between 0.0 and 1.0, indicating how well the evidence supports the verification_status)"
    }
  ],
  "overall_grounding_score": "float (the proportion of 'SUPPORTED' claims out of total claims, between 0.0 and 1.0)",
  "audit_summary": "string (a concise summary of the audit findings, highlighting any critical unsupported or contradicted claims)"
}

# CONSTRAINTS

1. **Atomicity**: Each claim must be a single, self-contained factual statement. Break down compound sentences. For example, "The product launched in 2020 and was updated in 2022" becomes two claims.
2. **Fidelity**: The `claim_text` must be a direct quote or a minimally rephrased statement from the [ANSWER]. Do not alter the meaning.
3. **Evidence Grounding**: For a claim to be 'SUPPORTED', you must find explicit evidence in [SOURCE_DOCUMENTS]. Inference is not support. If the evidence implies the claim but does not state it, the status is 'UNSUPPORTED'.
4. **Source Exclusivity**: Only use the provided [SOURCE_DOCUMENTS] for verification. Do not use external knowledge. If the evidence is silent on a claim, the status is 'NOT_ADDRESSED'.
5. **Strict JSON**: Your entire response must be the valid JSON object described in the OUTPUT SCHEMA. No markdown fences, no introductory text.

# RISK LEVEL
[RISK_LEVEL]

# ADDITIONAL INSTRUCTIONS
[CONSTRAINTS]

To adapt this template, start by defining your [ANSWER] and [SOURCE_DOCUMENTS] variables. The [SOURCE_DOCUMENTS] should be a pre-formatted string that includes clear source identifiers (e.g., [DOC_ID: 1]) before each chunk of text. The [RISK_LEVEL] placeholder can be used to inject context-specific instructions, such as "HIGH: If any claim is UNSUPPORTED or CONTRADICTED, the overall_grounding_score must be 0.0." The [CONSTRAINTS] field allows you to add domain-specific rules, like "For legal claims, require a direct quote from a statute or case law." The most critical adaptation is ensuring the [SOURCE_DOCUMENTS] are chunked and identified in a way that the model can reliably reference them in the source_id field. Test this with a small, known set of claims and documents before integrating it into a CI/CD pipeline.

This prompt is designed to operate as a post-generation verification gate within a CI/CD pipeline, not as a user-facing step. After your primary RAG system generates an answer, this prompt decomposes that answer into discrete factual claims and verifies each against the original retrieved evidence. The output is a structured audit trail—a list of claims, their grounding status, and per-claim scores—that your pipeline can use to block deployment, flag for review, or log for observability. The harness must treat this prompt as a deterministic evaluation step: same input should produce a comparable claim structure every time, which is why strict JSON output and schema validation are non-negotiable.

Integration pattern: Wire the prompt into an evaluation script that runs after answer generation but before the answer reaches a user or a release gate. The script should (1) receive the generated answer and the source context used to produce it, (2) call the model with this prompt, (3) parse the JSON output, (4) validate the schema, and (5) apply pass/fail rules. A minimal harness in Python might look like: claims = call_model(prompt_template.format(answer=generated_answer, context=retrieved_chunks)) followed by validate_claims_schema(claims) and assert all(c['grounding_score'] >= 0.7 for c in claims['claims']). The grounding score threshold is a tunable parameter—start at 0.7 and adjust based on your tolerance for unsupported claims. For high-stakes domains (legal, clinical, financial), require human review when any claim scores below 0.9 or when the unsupported_claims array is non-empty.

Validation and retry logic: The harness must validate the JSON structure before trusting the results. Check that every claim object contains claim_text, source_evidence_match, grounding_score, and verification_status fields. If the model returns malformed JSON, implement a retry with the same prompt plus the parsing error message appended as a correction hint. Limit retries to 2 attempts; after that, escalate the entire answer for human review rather than silently passing. Log every verification run—including the raw prompt, model response, parse errors, and final scores—to your observability platform. This log becomes your audit evidence when stakeholders ask why an answer was blocked or approved. For model choice, prefer models with strong JSON mode support (GPT-4o, Claude 3.5 Sonnet) and set temperature=0 to maximize output stability. Do not use this prompt with models that lack reliable structured output; the verification step is only as trustworthy as the schema compliance.

CI/CD gate integration: In your CI/CD pipeline, treat the grounding verification results as a quality gate. Define a policy such as: 'If any claim has grounding_score < 0.5 or verification_status == "unsupported", block the deployment and create a ticket.' For less critical applications, you might only block when more than 10% of claims are unsupported. Store the verification results alongside the answer in your evaluation database so you can track grounding quality over time and detect regressions when prompts or retrieval pipelines change. The claim_extraction_harness should also run against a golden dataset of known-good and known-bad answers to calibrate your thresholds before going live. Avoid the temptation to skip this harness in pre-production; an unverified RAG answer that reaches users is a hallucination incident waiting to happen.