Unsupported Claim Flagging Prompt

This prompt is a guardrail component for output validation pipelines. Its job is to analyze a generated text alongside a set of provided source documents and identify every factual claim that lacks supporting evidence. It distinguishes between claims that are unsupported because the sources are silent on the topic and claims that directly contradict the provided evidence. The output includes a severity classification for each flagged claim so that downstream systems can route low-risk stylistic statements differently from high-risk factual fabrications.

Use this prompt after answer generation and before the response reaches a user, auditor, or downstream system. It assumes you have already retrieved or provided the relevant source documents. This is not a retrieval prompt, not a citation formatter, and not a general fact-checker that searches the open web. The ideal user is an AI product engineer or output validation pipeline builder who needs a deterministic, auditable step between generation and delivery. Required context includes the full generated text to be verified and the complete set of source documents that should support it. Without both, the prompt cannot function—it will either flag everything as unsupported or miss fabrications entirely.

Do not use this prompt when you need to retrieve sources, when you need to format citations, or when you need a human-readable explanation of why a claim is supported. It is not a replacement for retrieval-augmented generation (RAG) and should not be used as the primary factuality check in systems where source documents are not available. If your workflow requires open-web verification, use a separate retrieval step before invoking this prompt. For high-risk domains such as healthcare, legal, or financial compliance, always route severity-flagged outputs to human review before any automated action is taken. The next step after reading this section is to review the prompt template and wire it into your validation harness with appropriate severity thresholds and escalation rules.

This prompt template is designed to be dropped directly into your output validation or guardrail step. It instructs the model to act as a strict evidence auditor: it must identify every factual claim in a generated text, check whether that claim is supported by the provided source material, and classify any unsupported claim by severity. The output is a structured report that your application can parse to decide whether to block, flag, or escalate the response before it reaches a user.

textCopy
You are an evidence auditor. Your task is to identify claims in the provided model output that lack support from the provided source documents.

## INPUT
Model Output:
[OUTPUT_TEXT]

Source Documents (with IDs):
[SOURCE_DOCUMENTS]

## INSTRUCTIONS
1. Extract every discrete factual claim from the Model Output. A claim is a statement that can be verified against evidence.
2. For each claim, search the Source Documents for direct supporting evidence.
3. Classify each claim as:
   - SUPPORTED: The claim is directly backed by at least one source passage.
   - UNSUPPORTED: No source provides evidence for the claim.
   - CONTRADICTED: A source explicitly states the opposite of the claim.
   - FABRICATED: The claim references a source, page, or detail that does not exist in the provided documents.
4. For each UNSUPPORTED, CONTRADICTED, or FABRICATED claim, assign a severity:
   - LOW: Minor detail, unlikely to mislead.
   - MEDIUM: Could cause confusion or minor factual error.
   - HIGH: Material misrepresentation, safety risk, or regulatory concern.
   - CRITICAL: Fabricated citation, invented data, or dangerous misinformation.

## OUTPUT SCHEMA
Return a JSON object with this structure:
{
  "total_claims": <integer>,
  "supported_count": <integer>,
  "unsupported_count": <integer>,
  "contradicted_count": <integer>,
  "fabricated_count": <integer>,
  "flagged_claims": [
    {
      "claim_text": "<exact claim from output>",
      "classification": "UNSUPPORTED | CONTRADICTED | FABRICATED",
      "severity": "LOW | MEDIUM | HIGH | CRITICAL",
      "reasoning": "<why this claim was flagged, referencing specific sources or gaps>"
    }
  ],
  "overall_assessment": "<one-sentence summary of output reliability>"
}

## CONSTRAINTS
- Do not flag claims that are clearly opinion, hedging, or stylistic language.
- If a claim is partially supported but overstates the evidence, classify it as UNSUPPORTED.
- If no source documents are provided, classify all factual claims as UNSUPPORTED.
- Never invent supporting evidence. If you are unsure, flag the claim.

To adapt this template, replace [OUTPUT_TEXT] with the model-generated text you are validating and [SOURCE_DOCUMENTS] with your retrieved or provided evidence, including source IDs for traceability. For high-risk domains such as healthcare or legal, add a [RISK_LEVEL] parameter that adjusts severity thresholds and always route CRITICAL and HIGH findings to a human review queue. Before deploying, run this prompt against a golden dataset of known supported and unsupported claims to calibrate your severity thresholds and measure precision and recall. The structured JSON output is designed to be consumed by downstream routing logic—parse the flagged_claims array and use the severity field to decide whether to block the response, surface a warning, or log for audit.

The Unsupported Claim Flagging Prompt is not a standalone tool; it is a guardrail component designed to sit inside a post-generation validation pipeline. After your primary model produces an answer, pass that answer and its source context into this prompt. The output is a structured JSON payload containing flagged claims, severity classifications, and evidence gap descriptions. This output should be consumed programmatically by your application logic—not displayed directly to end users. The typical integration point is after answer generation and before the response is returned to the user, or as an asynchronous audit step for compliance workflows.

To wire this into production, wrap the prompt call in a validation harness that handles model choice, retries, and structured output parsing. Use a fast, cost-efficient model for this task (such as GPT-4o-mini or Claude Haiku) since the verification logic is classification-heavy and does not require deep reasoning. Implement a retry loop with a maximum of 2 attempts: if the model returns malformed JSON or fails to produce the expected flagged_claims array, retry with the error message appended to the prompt. After parsing, validate the output schema—each flagged claim must contain claim_text, severity (one of unsupported, fabricated, contradicted), and evidence_gap_description. Reject outputs that do not conform. Log every verification result, including the input claims, the model's classification, and the final action taken, for auditability and prompt debugging.

For high-risk domains such as healthcare, legal, or finance, do not rely solely on this automated flag. Route any claim flagged with severity fabricated or contradicted to a human review queue before the response reaches the user. For unsupported claims, you may choose to either strip the claim from the output automatically or append a visible uncertainty marker, depending on your product's risk tolerance. Avoid the temptation to use this prompt as a real-time correctness guarantee—it reduces hallucination risk but does not eliminate it. Pair it with citation verification prompts and evidence sufficiency checks for a layered defense. The next step is to build an eval harness using golden datasets of known supported and unsupported claims to measure your pipeline's precision and recall before deployment.