Post-Generation Fact Audit Prompt

This prompt is designed for observability pipelines and production monitoring systems that need to audit a complete model response against its source context. The primary job-to-be-done is automated, systematic verification: you have a generated answer and the exact context it was supposed to use, and you need a machine-readable report confirming which claims are supported, which are fabricated, and how severe each finding is. The ideal user is an AI reliability engineer, a platform developer, or an MLOps practitioner embedding this audit as a post-generation gate before a response reaches a user or a compliance log. You should use this prompt when the cost of an undetected hallucination is high—such as in regulated industries, customer-facing accuracy guarantees, or internal decision-support tools—and when you can tolerate the latency of an asynchronous audit step.

To use this prompt effectively, you must provide two concrete inputs: the full model response to audit, and the complete source context that was available to the model at generation time. The prompt performs a claim-by-claim decomposition, mapping each atomic statement in the response back to the evidence. It then produces a structured output including a hallucination rate, a severity classification for each unsupported claim (e.g., minor, major, critical), and specific correction targets. This is not a real-time correction loop; it is a post-hoc audit. Do not use this prompt inside a synchronous user-facing request path where sub-second latency is required. Instead, run it asynchronously, log the results, and trigger alerts or downstream correction workflows based on the audit findings.

There are clear boundaries where this prompt is the wrong tool. If you need to prevent hallucination at generation time rather than detect it afterward, use a Context-Only Answer Constraint Prompt instead. If you need to immediately correct a flagged response and return it to a user, pair this audit with a Grounded Answer Regeneration Prompt in a retry harness. If your source context is a large, multi-document retrieval set with conflicting information, consider running a Source Conflict Resolution Prompt first, then auditing the reconciled answer. Finally, if you lack a reliable ground-truth context—for example, auditing a model's free-form creative writing or open-domain knowledge—this prompt will produce unreliable results because it cannot distinguish between correct world knowledge and hallucination without a provided evidence baseline. In high-risk domains such as healthcare, finance, or legal, always route audit findings through a Human-in-the-Loop Fact Review Prompt before taking action on the corrections.

This template is designed to be wired into a post-generation validation harness. After your primary model produces a response, this prompt instructs a secondary audit model to systematically cross-check every factual claim against the provided source context. The output is a structured JSON report that includes a hallucination rate, severity classifications, and specific correction targets, making it directly ingestible by monitoring dashboards, alerting systems, or downstream repair loops.

textCopy
You are a meticulous fact-auditor. Your task is to analyze a model-generated response against a provided source context. Perform a systematic, claim-by-claim audit and produce a structured report.

## INPUT
**Source Context:**
[SOURCE_CONTEXT]

**Model Response to Audit:**
[MODEL_RESPONSE]

## AUDIT INSTRUCTIONS
1. **Extract Claims:** Decompose the Model Response into a list of discrete, verifiable factual claims. Ignore opinions, stylistic phrasing, and purely structural sentences.
2. **Verify Each Claim:** For each claim, determine its relationship to the Source Context:
   - **SUPPORTED:** The claim is directly stated in or can be trivially inferred from the Source Context.
   - **UNSUPPORTED:** The claim introduces new information not found in the Source Context.
   - **CONTRADICTED:** The claim directly conflicts with information in the Source Context.
3. **Classify Severity:** For UNSUPPORTED and CONTRADICTED claims, assign a severity:
   - **CRITICAL:** The claim changes a core fact, entity, number, or timeline.
   - **MINOR:** The claim is a non-material detail or plausible but unverified elaboration.
4. **Calculate Hallucination Rate:** (Number of UNSUPPORTED claims + Number of CONTRADICTED claims) / Total Number of Claims.

## CONSTRAINTS
- Do not use external knowledge. Base your audit strictly on the provided Source Context.
- If the Source Context is empty or insufficient to verify a claim, classify it as UNSUPPORTED with a note.
- If the Model Response is empty or contains no factual claims, report a hallucination rate of 0 and an empty claims list.

## OUTPUT_SCHEMA
You must respond with a single JSON object conforming to this structure:
{
  "audit_result": {
    "total_claims": <integer>,
    "supported_claims": <integer>,
    "unsupported_claims": <integer>,
    "contradicted_claims": <integer>,
    "hallucination_rate": <float between 0.0 and 1.0>,
    "claims": [
      {
        "claim_text": "<exact text of the claim from the response>",
        "verdict": "SUPPORTED | UNSUPPORTED | CONTRADICTED",
        "severity": "NONE | CRITICAL | MINOR",
        "source_evidence": "<exact quote from Source Context or 'NONE FOUND'>",
        "correction_target": "<suggested correction or null if SUPPORTED>"
      }
    ],
    "summary": "<one-sentence summary of the audit findings>"
  }
}

To adapt this template, replace [SOURCE_CONTEXT] with the retrieved documents or ground-truth data provided to the primary model, and [MODEL_RESPONSE] with the text you need to audit. For high-risk domains, ensure the OUTPUT_SCHEMA is validated by your application layer before the report is acted upon. If the audit model itself produces malformed JSON, use a separate output repair prompt to fix the structure before parsing. For production observability, log the hallucination_rate and severity counts as metrics, and trigger an alert if the rate exceeds a predefined threshold, such as 0.1.

The Post-Generation Fact Audit Prompt is designed to operate as a post-hoc validation step within an AI observability pipeline, not as a real-time user-facing feature. It should be invoked after a primary model response is generated and before the response is delivered to the user or logged as final. The harness must provide the audit prompt with the complete original response and the exact source context that was available to the primary model. Without strict context isolation—ensuring the audit model sees only the same evidence the primary model had—the audit loses its diagnostic value and may flag correct external knowledge as hallucination.

Wire the prompt into an asynchronous validation worker that consumes from a response queue. The worker should: (1) deserialize the primary response and its grounding context; (2) format the audit prompt with [RESPONSE], [CONTEXT], and [AUDIT_SCHEMA]; (3) call a separate, potentially cheaper or faster model for the audit step; (4) parse the structured audit report; and (5) write the audit result to an observability store alongside the original response. Implement a retry budget of 2 for audit parsing failures—if the audit model returns malformed JSON, retry with a stricter schema instruction before escalating. Log both the raw audit output and the parsed report for traceability.

For production monitoring, attach the audit report's hallucination_rate and severity_distribution fields as metrics to your existing observability platform (e.g., Datadog, Grafana, or a custom dashboard). Set alert thresholds based on rolling windows: a spike in critical severity claims or a sustained hallucination_rate above a defined baseline should trigger an on-call notification. Store the per-claim audit details in a queryable log for offline pattern analysis. Do not use the audit prompt to automatically block or rewrite responses unless you have a human-in-the-loop review step for high-severity findings; the audit model itself can hallucinate, and automated suppression based on a second model's judgment creates a new failure mode. Instead, use the audit output to prioritize manual review queues and feed correction examples back into your primary prompt engineering cycle.