Post-Generation Factual Audit Prompt Template

This prompt is designed for a single, high-stakes job: verifying a RAG-generated answer against its source context after the answer has been produced but before it reaches the end user. The ideal user is a QA engineer, platform engineer, or AI architect implementing an automated release gate in a production pipeline. The required context is a complete, unmodified set of source documents or chunks that were used to generate the answer, alongside the final answer text itself. The prompt assumes the answer already exists; it does not assess whether the initial retrieval was sufficient, nor does it rewrite the answer to fix errors. Its sole purpose is to produce a structured audit report that flags unsupported statements, identifies missing citations, and detects factual contradictions, enabling a programmatic pass/fail decision.

Use this prompt when factual accuracy is non-negotiable and the cost of a hallucination reaching a user is high—for example, in clinical decision support, legal research, financial analysis, or compliance documentation. It is particularly effective when you need a consistent, machine-readable verdict that can trigger automated actions: blocking a response, routing it for human review, or logging an incident. The prompt works best with capable instruction-following models (e.g., GPT-4, Claude 3.5 Sonnet) that can perform fine-grained textual alignment. You should wire the output into a validation layer that checks for the required JSON schema and retries on parse failures. For high-risk domains, always include a human review step for any answer that receives a FAIL verdict, and log the full audit report alongside the answer for auditability.

Do not use this prompt as a pre-generation evidence sufficiency check—that requires a different prompt designed to assess whether the retrieved context can answer the question at all. Do not use it to evaluate the stylistic quality, tone, or helpfulness of an answer; it is strictly a factual audit. Avoid using it on extremely long answers (e.g., multi-page reports) without first segmenting the answer into verifiable units, as the model's attention may degrade. If your primary concern is citation format compliance rather than factual accuracy, use a lighter-weight citation cross-reference prompt instead. Finally, remember that this prompt is a probabilistic guardrail, not a deterministic verifier—it reduces hallucination risk but does not eliminate it. Combine it with complementary checks like embedding-based semantic similarity and human sampling to build a defense-in-depth strategy.

The following prompt template is designed to be pasted directly into your audit harness. It instructs the model to act as a meticulous fact-checker, comparing a generated answer against the provided source context. The prompt is structured to produce a machine-readable JSON audit report, making it suitable for automated pass/fail gating in a CI/CD pipeline or a pre-release review step. Before using this template, ensure you have the generated answer, the exact retrieved context used to produce it, and the original user query.

textCopy
You are an expert fact-checker auditing a generated answer for factual accuracy. Your task is to compare the answer against the provided source context and produce a structured audit report.

## INPUT
**User Query:** [USER_QUERY]
**Source Context (retrieved documents):**
[SOURCE_CONTEXT]
**Generated Answer:**
[GENERATED_ANSWER]

## AUDIT INSTRUCTIONS
1.  **Decompose the Answer:** Break the generated answer down into discrete, verifiable factual claims.
2.  **Verify Each Claim:** For each claim, determine if it is **SUPPORTED**, **UNSUPPORTED**, or **CONTRADICTED** by the source context. A claim is SUPPORTED only if the context directly states it. A claim is UNSUPPORTED if the context is silent on the matter. A claim is CONTRADICTED if the context states the opposite.
3.  **Check for Citations:** If the answer includes citations, verify that each cited source genuinely supports the claim it is attached to. Flag any citation fabrication.
4.  **Identify Missing Information:** Note any part of the user query that the source context cannot answer, and check if the generated answer correctly abstains or acknowledges the gap.

## OUTPUT FORMAT
Return a single JSON object with the following schema:
{
  "audit_result": "PASS" | "FAIL",
  "overall_score": 0.0-1.0,
  "claims_audit": [
    {
      "claim_text": "string",
      "verdict": "SUPPORTED" | "UNSUPPORTED" | "CONTRADICTED",
      "source_evidence": "string or null",
      "citation_valid": true | false | "N/A"
    }
  ],
  "flagged_issues": [
    {
      "severity": "CRITICAL" | "MAJOR" | "MINOR",
      "type": "UNSUPPORTED_CLAIM" | "CONTRADICTION" | "CITATION_FABRICATION" | "MISSING_ABSTENTION",
      "description": "string"
    }
  ],
  "abstention_check": {
    "query_has_gap": true | false,
    "answer_abstained_correctly": true | false | "N/A",
    "explanation": "string"
  }
}

## CONSTRAINTS
- Base your audit strictly on the provided source context. Do not use outside knowledge.
- If the source context is insufficient to verify a claim, the verdict must be UNSUPPORTED.
- The overall_score should reflect the proportion of SUPPORTED claims, penalized heavily for CONTRADICTED claims.
- Set audit_result to "FAIL" if any CRITICAL or MAJOR issue is found.

To adapt this template for your specific pipeline, replace the square-bracket placeholders with your actual data. The [SOURCE_CONTEXT] should be the exact text of the retrieved chunks, including any document IDs or metadata you want the model to reference. For high-stakes domains like healthcare or finance, you should add a [RISK_LEVEL] parameter and adjust the CONSTRAINTS section to require a higher evidence threshold for a PASS verdict. The JSON output schema is designed to be parsed by downstream validation code, which can enforce your team's specific pass/fail rules, such as a minimum overall_score or a zero-tolerance policy for CONTRADICTED claims.

After pasting this prompt, the next critical step is to validate the model's output against the expected schema. A malformed JSON response from the audit step should trigger a retry or a safe failure. For production systems, log every audit report alongside the answer and context to build a dataset for refining your RAG pipeline and your audit criteria. Do not treat this prompt as a substitute for human review in high-risk applications; instead, use it as a scalable first-pass filter to catch obvious hallucinations before they reach a human reviewer or end-user.

The Post-Generation Factual Audit prompt is designed to operate as a synchronous gate in your answer delivery pipeline. After the primary RAG system generates an answer, the raw output, the retrieved context passages, and the user's original query are all passed to this audit prompt before the answer reaches the user. The audit prompt's structured output—containing a pass/fail status, a list of flagged claims, and a severity breakdown—is parsed by your application to decide whether to release the answer, block it, or escalate for human review. Do not treat this as an offline evaluation step; it is a runtime guardrail that must complete within your latency budget.

To wire this into production, wrap the model call in a validation and retry layer. First, parse the model's JSON output against a strict schema that expects fields like audit_status, flagged_claims, and overall_severity. If the JSON is malformed or missing required fields, use an Output Repair prompt to fix the structure before proceeding. If the audit status is pass, release the answer. If the status is fail with a severity of critical or high, block the answer and log the full audit report for immediate review. For medium severity failures, you may choose to surface the answer with a caveat, but only if your product's risk tolerance explicitly allows it. Implement a retry budget of no more than two attempts: if the audit prompt itself fails to produce valid JSON after two repair attempts, escalate the entire request to a human review queue rather than silently passing a potentially flawed answer.

For teams running at scale, consider model routing for this audit step. A smaller, faster model can perform the initial claim extraction and grounding check, while a more capable model is reserved for ambiguous cases where the smaller model's confidence score falls below a defined threshold (e.g., below 0.85). Log every audit result—pass and fail—along with the prompt version, model identifier, and timestamp. These logs become your audit trail for governance and your dataset for improving the audit prompt itself. Avoid the temptation to skip the audit when latency spikes; instead, invest in prompt caching for the system instructions and source passages that remain stable across requests.