Zero-Evidence Claim Flagging Prompt Template

This prompt is a first-pass, high-recall filter designed to catch factual claims in a generated answer that have absolutely no supporting evidence in the provided source context. Its primary job is to act as a cheap, fast gate before more expensive verification steps like claim-by-claim source matching or human review. The ideal user is a QA engineer, eval pipeline builder, or AI architect who needs to automate hallucination detection in a RAG system and is willing to tolerate some false positives in exchange for catching nearly every unsupported claim. The required context is a generated answer and the exact set of source passages that were provided to the generation model.

Use this prompt when you need a lightweight safety net that runs after answer generation but before the answer reaches the user. It is particularly effective in high-volume pipelines where running detailed verification on every output would be cost-prohibitive. The prompt works best when the source context is well-chunked and the generated answer is self-contained rather than conversational. Wire it into automated guardrails that flag outputs for review, block them from delivery, or route them to a secondary verification step. Do not use this prompt as your only hallucination defense in high-stakes domains like healthcare, legal, or finance—always pair it with claim-by-claim verification and human review for those use cases.

This prompt is not designed to detect subtle hallucinations where the model paraphrases evidence in a misleading way, nor does it catch claims that are partially supported but distorted. It also cannot assess whether the retrieved context itself is sufficient to answer the user's question—that requires a separate evidence sufficiency gate. If you need fine-grained grounding checks, use the Sentence-Level Grounding Check Prompt instead. If you need to verify that citations genuinely support their attached claims, use the Citation-to-Evidence Cross-Reference Prompt. The next step after flagging zero-evidence claims is typically to route flagged outputs to a more detailed verification prompt or a human review queue, depending on your risk tolerance and operational budget.

This prompt template is designed to be the first-pass filter in a hallucination detection pipeline. Its job is high-recall identification of statements that have no grounding whatsoever in the provided source material. It does not assess partial support, contradiction, or interpretation quality—those are left to more expensive, downstream verification steps. The template expects a generated answer and the exact context that was used to produce it. It returns a structured list of zero-evidence claims with their exact text spans, making the output directly consumable by automated guardrails, logging systems, or human review queues.

textCopy
You are a strict factual auditor. Your task is to identify every claim in the [ANSWER] that has zero supporting evidence in the [CONTEXT]. A claim is a factual assertion that a reasonable reader would understand as a statement about the world. Ignore opinions, greetings, disclaimers, and purely stylistic language.

For each claim in the answer, check whether the context provides any information that directly supports it. If the context is silent on the claim, the claim is unsupported. If the context provides information that could be interpreted to support the claim only with additional inference or assumption, treat it as unsupported. Only mark a claim as supported if the context explicitly contains the same factual information.

Return a JSON object with the following schema:
{
  "zero_evidence_claims": [
    {
      "claim_text": "exact text span from the answer",
      "claim_start_char": integer,
      "claim_end_char": integer,
      "reason": "brief explanation of why no evidence was found"
    }
  ],
  "total_claims_checked": integer,
  "total_zero_evidence": integer
}

If no claims lack evidence, return an empty list for zero_evidence_claims and set total_zero_evidence to 0.

[ANSWER]
{answer}

[CONTEXT]
{context}

To adapt this template, replace {answer} with the generated text you are auditing and {context} with the retrieved passages that were available during generation. If your context has metadata such as document IDs or chunk indices, include it in the context block so the model can reference it in its reasoning. For high-stakes domains, add a [RISK_LEVEL] parameter that adjusts the strictness of the evidence requirement—at high, even implicit support should be flagged. Always run this prompt before more expensive verification steps; it is designed to be fast and high-recall, catching obvious fabrications so downstream classifiers only need to handle ambiguous cases.

The Zero-Evidence Claim Flagging prompt operates as a first-pass filter in a multi-stage hallucination detection pipeline. Its job is high-recall, not high-precision: it should catch every claim that lacks any supporting evidence, even at the cost of some false positives. This means the implementation harness must treat flagged claims as candidates for further verification, not as final verdicts. Wire this prompt after answer generation and before more expensive verification steps like claim-by-clause source matching or human review. The typical flow is: RAG answer generation → zero-evidence flagging → (optional) detailed claim verification → human review queue for unresolved flags.

Input assembly requires three fields: the generated answer text, the full retrieved context passages, and an optional [RISK_LEVEL] parameter that controls the strictness of flagging. For high-risk domains (healthcare, legal, finance), set [RISK_LEVEL] to strict to flag even borderline unsupported claims. For lower-stakes applications, use standard to reduce false positives. The context passages should include document IDs and chunk metadata so downstream systems can trace flagged claims back to their supposed sources. Model choice matters: use a model with strong instruction-following and structured output capabilities (GPT-4, Claude 3.5 Sonnet, or equivalent). Cheaper models often struggle with the precise span extraction this prompt requires, producing malformed JSON or missing claims.

Validation and retry logic must handle three failure modes. First, JSON parse failures: if the model returns malformed JSON, run a lightweight repair prompt that feeds the raw output and the expected schema back to the model with instructions to fix formatting only. Second, span mismatch failures: validate that each flagged claim's text_span actually appears in the original answer text using exact substring matching. If spans don't match, discard those flags and log the mismatch for prompt refinement. Third, empty context failures: if the context passages are empty or truncated, the prompt will flag everything—add a pre-check that verifies context length meets a minimum threshold before invoking the flagging prompt. Set a maximum of 2 retries before escalating to a human review queue with the raw output attached.

Logging and observability are critical because this prompt is a pipeline gate. Log every invocation with: the prompt version hash, model identifier, input token count, output token count, latency, number of claims flagged, and the raw JSON response. Attach these logs to the parent answer generation trace so operators can correlate flagging behavior with specific RAG outputs. For high-stakes deployments, add a human-in-the-loop step: if more than a configurable threshold of claims are flagged (e.g., 30% of sentences), automatically route the answer to a review queue rather than blocking it outright. This prevents the system from silently discarding answers that might still be useful with appropriate caveats.

What to avoid: Do not use this prompt as the sole hallucination gate before user delivery. It catches unsupported claims but does not verify whether cited evidence actually supports the claims it accompanies—that requires a separate Citation-to-Evidence Cross-Reference check. Do not skip span validation; models occasionally hallucinate the flagged text spans themselves, creating a meta-hallucination problem. Do not run this prompt on every user message in a chat session; instead, run it only on the final synthesized answer to control latency and cost. Finally, version this prompt like any other production code: store it in a prompt registry with change history, run regression tests against a golden set of known hallucinated answers, and gate deployment on passing eval thresholds for both recall (catching real hallucinations) and precision (not overwhelming downstream reviewers with false positives).