Unsupported Claim Flagging Prompt for Output Validation

This prompt is designed for AI quality engineers and output validation pipeline builders who need to verify that a generated answer is fully supported by provided evidence. It operates as a post-generation guardrail: after an LLM produces an answer, this prompt decomposes the answer into atomic claims, checks each claim against the evidence set, and flags any claim that lacks direct support. The output is a structured report with severity classifications, making it suitable for automated validation gates, audit trails, and monitoring dashboards. Use this when you need systematic claim-level grounding verification rather than a single holistic faithfulness score.

The prompt requires two concrete inputs: a candidate answer (the text to validate) and an evidence set (the source passages against which claims are checked). It does not perform retrieval itself—it assumes you have already retrieved or provided the evidence. This separation is critical: the prompt is a verification tool, not a retrieval quality fix. If your retrieval pipeline is returning irrelevant or noisy passages, this prompt will correctly flag high rates of unsupported claims, but it won't solve the underlying retrieval problem. Wire it downstream of your answer generation step, and feed it the same evidence that was used to produce the answer.

Do not use this prompt for real-time answer generation or as a substitute for retrieval quality improvements. It adds latency and token cost to your pipeline, so reserve it for high-stakes outputs where grounding verification is required—compliance reports, customer-facing answers in regulated domains, or any workflow where an unsupported claim could cause user harm or audit failure. For low-risk or high-throughput applications, consider lighter-weight alternatives such as a single holistic faithfulness score or spot-checking with a smaller sample of claims. When you do use this prompt, always pair it with a human review step for claims flagged as critical or high-severity, and log the structured output for audit trails and eval dashboards.

This prompt template is the core instruction set for an output validation guardrail. It instructs the model to act as a strict fact-checker, comparing every claim in a [GENERATED_ANSWER] against a set of [EVIDENCE_PASSAGES]. The goal is not to rewrite the answer, but to produce a structured report that flags unsupported, contradictory, or unverifiable statements. Use this template as the foundation for a post-generation validation step before any response reaches a user, especially in high-stakes domains like healthcare, finance, or legal tech.

textCopy
You are an expert fact-checker and output validator. Your task is to compare a generated answer against a set of provided evidence passages and flag every claim that is not fully supported.

## INPUT

**Generated Answer:**
[GENERATED_ANSWER]

**Evidence Passages:**
[EVIDENCE_PASSAGES]

## INSTRUCTIONS

1.  **Decompose:** Break the Generated Answer into discrete, verifiable claims. A claim is a single factual assertion.
2.  **Match:** For each claim, search the Evidence Passages for direct support. A claim is supported only if the evidence explicitly states the fact or it is a direct, inescapable logical consequence of the stated facts.
3.  **Classify:** Assign one of the following severity levels to each unsupported claim:
    *   **CRITICAL:** The claim contradicts the evidence.
    *   **MAJOR:** The claim introduces a new fact not present in the evidence.
    *   **MINOR:** The claim contains a speculative or interpretive statement presented as fact.
    *   **NEGLIGIBLE:** The claim is a trivial detail (e.g., a date format, a synonym) that doesn't change the factual meaning.
4.  **Explain:** For every flagged claim, provide a concise reason for the flag, citing the specific evidence (or lack thereof).

## OUTPUT FORMAT

Return a single JSON object with the following schema:
{
  "overall_grounding_score": <float 0.0-1.0>,
  "total_claims": <int>,
  "supported_claims": <int>,
  "flagged_claims": [
    {
      "claim_text": "<exact text from answer>",
      "severity": "<CRITICAL|MAJOR|MINOR|NEGLIGIBLE>",
      "reason": "<explanation of why it's unsupported>",
      "relevant_evidence_snippet": "<supporting or contradicting text from evidence, or 'NONE'>"
    }
  ]
}

## CONSTRAINTS

*   Do not flag claims that are direct quotes or accurate paraphrases of the evidence.
*   If the evidence is insufficient to verify a claim, flag it as MAJOR.
*   Do not invent evidence. If no evidence exists, state 'NONE'.
*   Your entire response must be the single JSON object. No other text.

To adapt this template, replace the [GENERATED_ANSWER] and [EVIDENCE_PASSAGES] placeholders with your actual data. For production use, consider adding a [RISK_LEVEL] parameter to adjust the strictness of the classification logic. For example, in a high-risk context, you might reclassify all MINOR flags as MAJOR. The output schema is designed for programmatic consumption; a downstream rule engine can use the overall_grounding_score and the count of CRITICAL flags to decide whether to block the response, send it for human review, or allow it with a caveat. Always pair this prompt with an evaluation harness that measures flag recall and false-positive rate against a golden dataset of known supported and unsupported claims.

This prompt is designed to operate as a post-generation guardrail, not a real-time user-facing step. In a typical architecture, a primary model generates an answer from retrieved evidence, and that answer-evidence pair is then passed to this flagging prompt. The flagging prompt should be called by a validation service or an orchestration layer after the primary generation but before the answer is released to the user. This separation of concerns keeps the flagging logic independent, testable, and swappable without touching the primary generation prompt.

The implementation harness must handle structured input assembly and output parsing. The prompt expects two required inputs: the generated answer text and the source evidence passages. These should be injected into the [GENERATED_ANSWER] and [SOURCE_EVIDENCE] placeholders. The output is a JSON object containing an array of flagged claims, each with a claim text, severity classification, and a rationale. The harness must parse this JSON, validate it against a defined schema (e.g., using Pydantic or JSON Schema), and handle malformed responses with a retry or fallback. A critical implementation detail is claim segmentation: the prompt works best when the generated answer is relatively short (under 500 words). For longer answers, pre-segment the text into paragraphs or sentences and run the flagging prompt per segment to avoid the model missing claims.

The severity classification field drives downstream actions. Map the output severity levels to automated decisions: critical flags should block the response and escalate for human review; major flags should trigger a regeneration attempt with stricter grounding instructions; minor flags can be logged and attached as metadata for monitoring without blocking the response. Log every flagging result, including the full prompt, the model's raw output, and the parsed flags, to enable offline evaluation of false-positive and false-negative rates. For high-stakes domains, always require a human-in-the-loop review step for any response that triggers a critical or major flag before the answer reaches the end user. Do not use this prompt as the sole safety mechanism; combine it with upstream evidence sufficiency checks and downstream user-facing disclaimers.