Hallucination Risk Flagging Prompt

This prompt is designed for RAG and generation quality teams who need to systematically identify unsupported claims in model outputs before they reach end users. The primary job-to-be-done is automated hallucination detection: given a piece of generated text and its source evidence, the prompt flags specific segments that contain claims not grounded in the provided context. The ideal user is an AI engineer or quality lead building a production guardrail, not an end user casually checking a single response. You need the original generated output, the exact retrieval context or source documents used, and a predefined risk level taxonomy before you can use this prompt effectively.

Do not use this prompt when you lack access to the grounding evidence. It cannot detect hallucinations from the generated text alone—it requires source documents to compare against. It is also not a replacement for human review in high-stakes domains like healthcare or legal compliance, where unsupported claims carry regulatory risk. The prompt works best as a filter in an automated pipeline where flagged outputs are routed for human review or blocked entirely. It is not designed for real-time chat correction, creative writing evaluation, or fact-checking against world knowledge the model may possess outside the provided context.

Before deploying, define your risk level taxonomy clearly. A three-tier system such as GROUNDED, UNCERTAIN, and HALLUCINATION is common, but you may need finer gradations like PARTIALLY_SUPPORTED or CONTRADICTED. The prompt's value comes from consistent, constrained output that downstream systems can act on without parsing ambiguity. Pair this prompt with a validation layer that rejects any risk label outside your approved vocabulary. Start with a golden test set of known grounded and hallucinated examples to calibrate your thresholds before production use.

The following prompt template is designed to be copied directly into your prompt management system or codebase. It instructs the model to act as a strict hallucination auditor, comparing a generated output segment against a set of source documents. The model is forced to choose from a constrained vocabulary of risk levels, ensuring that downstream automation can reliably parse and act on the result without encountering unexpected labels.

textCopy
You are a strict hallucination auditor. Your task is to compare a generated [OUTPUT_SEGMENT] against the provided [GROUNDING_EVIDENCE] and assign a hallucination risk level.

You must classify the output into exactly one of the following risk levels:
- NO_RISK: All claims in the output are directly and unambiguously supported by the evidence.
- LOW_RISK: The output contains minor paraphrasing or summarization that does not alter the factual meaning.
- HIGH_RISK: The output contains a claim, entity, date, or figure that is not present in the evidence or directly contradicts it.
- UNVERIFIABLE: The output makes a claim that cannot be confirmed or denied using the provided evidence alone.

[GROUNDING_EVIDENCE]:
"""
[EVIDENCE_TEXT]
"""

[OUTPUT_SEGMENT]:
"""
[SEGMENT_TEXT]
"""

[CONSTRAINTS]:
- If the output is a direct quote from the evidence, it is NO_RISK.
- If the output synthesizes multiple pieces of evidence correctly, it is NO_RISK.
- If the output adds a plausible but unsupported detail (e.g., a date or name), it is HIGH_RISK.
- Do not use external knowledge. Base your decision strictly on the [GROUNDING_EVIDENCE] provided.

Return a single JSON object with the following keys:
- "risk_level": string (one of NO_RISK, LOW_RISK, HIGH_RISK, UNVERIFIABLE)
- "rationale": string (a brief, specific explanation citing the evidence or noting its absence)
- "flagged_span": string (the exact substring from the [OUTPUT_SEGMENT] that triggered the risk level, or null if NO_RISK)

To adapt this template, replace the [EVIDENCE_TEXT] and [SEGMENT_TEXT] placeholders with your actual data. For production use, ensure that the [OUTPUT_SCHEMA] is enforced by your application's parsing layer, not just by the prompt. If the model returns a risk_level outside the constrained set, the response should be rejected and retried. In high-stakes domains like healthcare or finance, a HIGH_RISK or UNVERIFIABLE flag must trigger a human review step before the output reaches an end user. Start by testing this prompt against a golden dataset of known hallucinations and factual statements to calibrate your tolerance for false positives and false negatives.

This prompt is designed to sit directly after a generation step in a RAG or content-creation pipeline. Its job is to receive a piece of generated text and its corresponding grounding evidence, then return a structured risk assessment. The implementation harness must guarantee that the evidence passed to the prompt is the exact set of documents or context chunks used during generation, not a post-hoc retrieval. Any mismatch between the generation context and the verification context will produce unreliable risk scores.

The harness should enforce a strict contract: the model's output must be a valid JSON object matching the defined schema, including the risk_level enum (LOW, MEDIUM, HIGH, CRITICAL) and a list of flagged_segments. Implement a validation layer that rejects any response that doesn't parse as JSON or contains an out-of-vocabulary risk_level. For CRITICAL or HIGH risk outputs, the harness should automatically route the content to a human review queue and block it from being served to an end-user. For MEDIUM risk, log the output and the flagged segments for offline audit without blocking the pipeline. Use a model with strong instruction-following and JSON mode capabilities (such as gpt-4o or claude-3.5-sonnet) and set temperature=0 to maximize deterministic, repeatable evaluations.

To prevent silent failures, build an evaluation harness that runs this prompt against a golden dataset of generated statements paired with evidence. The dataset must include clear-cut factual statements (expecting LOW risk), statements with minor inferred details (expecting MEDIUM), and completely unsupported claims (expecting HIGH or CRITICAL). Track false negatives—where an unsupported claim is marked LOW—as a critical product metric. A common failure mode is the model accepting a plausible-sounding but unsupported statement as factual because it aligns with the model's own pre-training knowledge. Mitigate this by explicitly instructing the model to only consider the provided evidence, and run regular adversarial tests with statements that are true in the world but absent from the provided context.

For high-throughput systems, consider batching multiple statement-evidence pairs into a single prompt call to reduce API overhead, but be aware that longer contexts can degrade the model's attention to each individual pair. Implement a retry layer with exponential backoff for API failures, but do not retry on validation failures—a malformed output should be logged as an error and escalated immediately. Finally, store every risk assessment alongside the generation ID, evidence hash, and model version to create an audit trail that supports debugging, compliance reviews, and prompt regression testing over time.