Automated Guardrail Prompt for Hallucination Detection

This prompt is a reusable system-level guardrail that sits between your answer generation step and the end user. It inspects a generated answer against the provided source evidence and returns a structured decision: block, flag for review, or pass. Platform teams use this to build centralized safety infrastructure that catches hallucinations, policy violations, and unsupported claims before they reach customers. Deploy this when you need a single, configurable checkpoint that enforces factual grounding across multiple RAG applications or product surfaces. This is not a prompt for generating answers. It is a prompt for verifying them.

The ideal user is a platform engineer or AI safety architect responsible for output quality across multiple product surfaces. You need the generated answer, the source evidence used to produce it, and a defined policy for what constitutes a violation. The guardrail prompt works best when you can supply a clear [POLICY] block that defines unacceptable behaviors—such as unsupported claims, fabricated citations, contradictions with source material, or statements that exceed the evidence. You should also supply an [OUTPUT_SCHEMA] so the guardrail returns a machine-readable decision your application can act on without parsing free text. For high-risk domains, configure the guardrail to flag for human review rather than auto-blocking, and log every decision with the guardrail's rationale for audit.

Do not use this prompt as your only safety measure for high-stakes regulated domains. It is a probabilistic check, not a deterministic validator. Combine it with citation-to-evidence cross-reference checks, human review for flagged outputs, and regression tests against known hallucination patterns. Avoid using this guardrail on very short or trivial answers where the overhead of verification exceeds the risk. Also avoid deploying it without first calibrating the [POLICY] against real production examples—an overly strict policy creates a noisy flag queue that operators will ignore, while an overly permissive policy lets hallucinations through. Start with a policy tuned to catch clear violations, measure the flag rate and accuracy on production traffic, then tighten iteratively.

This prompt template is designed to be deployed as a system-level instruction for a secondary LLM call that sits between answer generation and user delivery. Its job is to act as a strict factual auditor: it receives the original user question, the generated answer, and the source evidence used to produce that answer, then returns a structured JSON decision. The guardrail does not rewrite the answer; it only decides whether the answer is supported, contains fabrications, or should be blocked based on configurable policies.

textCopy
You are a strict factual auditor. Your only job is to compare a generated answer against provided source evidence and return a structured JSON decision. You do not answer the user question. You do not improve the answer. You only audit.

## INPUT
- User Question: [USER_QUESTION]
- Generated Answer: [GENERATED_ANSWER]
- Source Evidence: [SOURCE_EVIDENCE]

## AUDIT RULES
1. Every factual claim in the Generated Answer must be directly supported by the Source Evidence. If a claim cannot be traced to a specific source passage, flag it as UNSUPPORTED.
2. If the Generated Answer contradicts the Source Evidence, flag the contradiction explicitly.
3. If the Generated Answer introduces entities, numbers, dates, names, or events not present in the Source Evidence, flag them as FABRICATED.
4. If the Generated Answer makes claims that go beyond what the Source Evidence supports (inference, speculation, extrapolation), flag them as UNSUPPORTED_INFERENCE.
5. Citations in the Generated Answer that point to sources that do not contain the cited claim are CITATION_FABRICATION.
6. If the Source Evidence is insufficient to answer the User Question at all, and the Generated Answer attempts to answer anyway, flag as INSUFFICIENT_EVIDENCE.
7. Policy violations in the Generated Answer (disallowed content, unsafe advice, regulated claims without disclaimers) must be flagged as POLICY_VIOLATION regardless of source support.

## OUTPUT SCHEMA
Return ONLY valid JSON with this exact structure:
{
  "decision": "PASS" | "FLAG" | "BLOCK",
  "decision_rationale": "One-sentence explanation of the decision.",
  "findings": [
    {
      "claim": "The exact claim text from the Generated Answer.",
      "claim_type": "SUPPORTED" | "UNSUPPORTED" | "FABRICATED" | "UNSUPPORTED_INFERENCE" | "CITATION_FABRICATION" | "CONTRADICTION" | "POLICY_VIOLATION",
      "source_reference": "The supporting or contradicting source passage, or null if none found.",
      "severity": "CRITICAL" | "HIGH" | "MEDIUM" | "LOW",
      "explanation": "Why this claim received this classification."
    }
  ],
  "block_reason": "If decision is BLOCK, the primary reason. Otherwise null.",
  "requires_human_review": true | false
}

## SEVERITY GUIDELINES
- CRITICAL: Fabricated safety instructions, medical advice, legal claims, financial figures, or personally identifiable information.
- HIGH: Fabricated facts that would mislead a reasonable user, unsupported statistics, or policy violations.
- MEDIUM: Minor unsupported details that do not change the core answer meaning.
- LOW: Stylistic embellishments or phrasing that does not introduce new factual content.

## DECISION RULES
- PASS: All claims are SUPPORTED. No findings of UNSUPPORTED, FABRICATED, or CONTRADICTION. Return empty findings array.
- FLAG: One or more MEDIUM or LOW severity findings exist. Answer may be usable with warnings. Set requires_human_review to false unless a HIGH finding also exists.
- BLOCK: Any CRITICAL or HIGH severity finding, any POLICY_VIOLATION, or INSUFFICIENT_EVIDENCE. Set requires_human_review to true.

## CONSTRAINTS
- Do not hallucinate source passages. If no source supports a claim, source_reference must be null.
- Do not re-interpret the Source Evidence. Take it literally.
- If the Generated Answer is empty or nonsensical, return BLOCK with decision_rationale explaining why.
- Never output anything except the JSON object.

To adapt this template for your application, replace the square-bracket placeholders with your actual runtime values. [USER_QUESTION] should contain the original user query. [GENERATED_ANSWER] is the full text output from your primary RAG or generation model. [SOURCE_EVIDENCE] should be the concatenated retrieved passages with their document identifiers and chunk references preserved so the auditor can cite them precisely. If your application has additional policies beyond factual accuracy—such as tone requirements, disclosure mandates, or domain-specific disclaimers—add them to the AUDIT RULES section. The severity guidelines should be calibrated to your risk tolerance: a fabricated product price in e-commerce may be HIGH, while the same in a clinical setting would be CRITICAL. Test the adapted prompt against a golden dataset of known hallucinations and correct answers before deploying it as a production guardrail. The structured JSON output is designed to be consumed by an upstream router that can either pass the answer to the user, attach a warning flag, or block delivery and escalate for human review.

This guardrail sits as a synchronous post-processing step between answer generation and user delivery. The generated answer and its source context are passed to the guardrail prompt, which returns a structured JSON decision (block, flag, or pass) along with a reason and a list of specific violations. The application layer must parse this decision and route accordingly: block prevents the answer from reaching the user and triggers an escalation or fallback; flag allows delivery but surfaces warnings to a review queue or end-user UI; pass permits normal delivery. The guardrail should never modify the answer—only inspect and decide.

Wire the prompt into a dedicated service or middleware function that accepts [ANSWER] and [CONTEXT] as inputs. Use a model with strong instruction-following and JSON output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet) and set response_format to json_object with a strict schema. Implement a validation layer that checks the returned JSON against the expected schema before acting on the decision. If the guardrail model returns malformed JSON or a decision outside the allowed enum (block, flag, pass), retry once with the same inputs and a stronger format reminder. After a second failure, log the raw output and escalate to a human review queue—never default to pass on parse failure in high-stakes domains. For latency-sensitive applications, run the guardrail in parallel with other post-generation checks (citation validation, PII redaction) and aggregate decisions before routing.

Instrument the guardrail with structured logging: capture the decision, violation list, model used, latency, and a hash of the input context for traceability. Store guardrail decisions alongside the answer in your observability platform to monitor hallucination rates over time and detect model drift. For regulated domains, ensure every block or flag decision is reviewable by a human within your defined SLA. Avoid using the guardrail as a substitute for retrieval quality improvements—if block rates exceed 5-10%, investigate your retrieval pipeline and evidence sufficiency before tuning the guardrail threshold. The guardrail is a safety net, not a primary quality control mechanism.