Hallucination Rate Comparison Prompt for Factual QA

This prompt is a regression measurement instrument, not a general answer-quality scorer. Use it when you are promoting a new prompt version, swapping a retrieval pipeline, or changing a model and need an evidence-backed gate before production. It compares two sets of factual QA outputs—a baseline and a candidate—against a shared set of source documents, producing a claim-level verification report that breaks down hallucination rates, severity, and types. The ideal user is a knowledge base team lead, RAG system builder, or AI quality engineer who already has a golden QA dataset, corresponding source documents, and both baseline and candidate answer sets ready for comparison.

Do not use this prompt for ad-hoc answer evaluation, single-turn debugging, or workflows where you lack a stable baseline. It assumes you have already run your golden QA dataset through both the old and new system configurations and collected the outputs. The prompt requires structured inputs: a list of QA pairs with both answers, plus the source documents that should ground the answers. Without these, the comparison cannot produce a reliable hallucination rate delta. This prompt also does not judge answer helpfulness, tone, or completeness—it focuses narrowly on factual grounding and hallucination detection.

Before running this prompt, ensure your golden QA dataset covers the failure modes you care about: ambiguous questions, multi-hop reasoning, conflicting sources, and edge cases where the model should abstain. A hallucination regression test is only as strong as the dataset it runs against. After executing the prompt, treat the output as a gate metric: if the candidate hallucination rate exceeds the baseline by a pre-defined threshold, block the release. Pair this prompt with a human review step for any severity-3 hallucinations (fully fabricated claims) flagged in the report.

The template below is designed to be dropped into an evaluation harness that feeds it pairs of outputs—one from a baseline prompt and one from a candidate prompt—along with the source documents used to generate them. It instructs the model to extract atomic claims from each output, verify each claim against the provided evidence, and produce a structured comparison report. The square-bracket placeholders represent the variables your application must supply at runtime: the two outputs to compare, the source context, the output schema you expect back, and any risk-level or domain-specific constraints.

codeCopy
You are a factual consistency auditor. Your task is to compare two AI-generated answers against the same set of source documents and measure the hallucination rate for each.

## INPUTS

### Source Documents
[DOCUMENTS]

### Baseline Answer (Version A)
[BASELINE_OUTPUT]

### Candidate Answer (Version B)
[CANDIDATE_OUTPUT]

## INSTRUCTIONS

1. Extract every atomic factual claim from the Baseline Answer. An atomic claim is a single verifiable statement that can be checked against the source documents.
2. Repeat for the Candidate Answer.
3. For each claim in both answers, determine its verification status:
   - **SUPPORTED**: The claim is directly stated in or logically entailed by the source documents.
   - **UNSUPPORTED**: The claim cannot be verified from the source documents. This is a hallucination.
   - **CONTRADICTED**: The claim directly conflicts with the source documents. This is a severe hallucination.
   - **OUT_OF_SCOPE**: The claim addresses a topic not covered by the source documents at all.
4. Classify each hallucination by type:
   - **FABRICATION**: Invented facts, numbers, names, or events not in sources.
   - **DISTORTION**: Misrepresented or exaggerated source information.
   - **OMISSION_AS_FACT**: Stating something as fact when the source only suggests or implies it.
   - **CITATION_ERROR**: Attributing a claim to the wrong source or section.
5. For every claim, cite the exact source passage(s) that support or contradict it. If no source passage exists, state "No supporting evidence found."

## OUTPUT FORMAT

Return a JSON object matching this schema:

{
  "baseline_analysis": {
    "total_claims": <integer>,
    "supported_claims": <integer>,
    "unsupported_claims": <integer>,
    "contradicted_claims": <integer>,
    "out_of_scope_claims": <integer>,
    "hallucination_rate": <float between 0.0 and 1.0>,
    "claims": [
      {
        "claim_text": "<exact claim from output>",
        "status": "SUPPORTED|UNSUPPORTED|CONTRADICTED|OUT_OF_SCOPE",
        "hallucination_type": "FABRICATION|DISTORTION|OMISSION_AS_FACT|CITATION_ERROR|null",
        "source_citation": "<verbatim source passage or 'No supporting evidence found.'>",
        "severity": "LOW|MEDIUM|HIGH|CRITICAL"
      }
    ]
  },
  "candidate_analysis": {
    "total_claims": <integer>,
    "supported_claims": <integer>,
    "unsupported_claims": <integer>,
    "contradicted_claims": <integer>,
    "out_of_scope_claims": <integer>,
    "hallucination_rate": <float between 0.0 and 1.0>,
    "claims": [
      {
        "claim_text": "<exact claim from output>",
        "status": "SUPPORTED|UNSUPPORTED|CONTRADICTED|OUT_OF_SCOPE",
        "hallucination_type": "FABRICATION|DISTORTION|OMISSION_AS_FACT|CITATION_ERROR|null",
        "source_citation": "<verbatim source passage or 'No supporting evidence found.'>",
        "severity": "LOW|MEDIUM|HIGH|CRITICAL"
      }
    ]
  },
  "comparison_summary": {
    "baseline_hallucination_rate": <float>,
    "candidate_hallucination_rate": <float>,
    "rate_change": <float, positive means candidate is worse>,
    "new_hallucination_types_introduced": ["<type>", ...],
    "severity_distribution_shift": "<description of how severity profile changed>",
    "regression_detected": <boolean>,
    "regression_details": "<explanation if regression detected, otherwise null>"
  }
}

## SEVERITY GUIDELINES
- **CRITICAL**: Contradicted claim on a safety-relevant, legal, or medically significant fact.
- **HIGH**: Fabricated specific numbers, names, dates, or statistics.
- **MEDIUM**: Distorted or exaggerated claim that changes meaning.
- **LOW**: Minor imprecision or out-of-scope claim with no material impact.

## CONSTRAINTS
- Do not invent source passages. If no evidence exists, say so explicitly.
- Treat the source documents as the only ground truth. Ignore your own knowledge.
- If [RISK_LEVEL] is HIGH or CRITICAL, flag every unsupported claim for human review.
- Count each atomic claim exactly once. Do not double-count.
- If an answer contains no factual claims (e.g., pure opinion or procedural instruction), set total_claims to 0 and hallucination_rate to null.

To adapt this template, replace [DOCUMENTS] with the retrieved passages or knowledge base entries that both outputs were generated from. Replace [BASELINE_OUTPUT] with the answer from your current production prompt, and [CANDIDATE_OUTPUT] with the answer from the prompt version you are testing. The [RISK_LEVEL] placeholder should be set to LOW, MEDIUM, HIGH, or CRITICAL based on your domain—this controls whether unsupported claims trigger a human-review flag in the output. If you are running this prompt inside an automated CI/CD pipeline, parse the regression_detected boolean and rate_change float from comparison_summary to decide whether to block promotion. For high-stakes domains like healthcare or legal, always route CRITICAL and HIGH severity claims to a human reviewer before accepting the automated comparison.

Before deploying this prompt into a regression suite, test it against a small golden set of known hallucinated and non-hallucinated output pairs. Verify that the model correctly identifies fabricated claims you have intentionally inserted and does not flag supported claims as hallucinations. If the model consistently misclassifies a particular hallucination type—such as treating omissions as supported—add a few-shot example to the prompt or tighten the definition in the INSTRUCTIONS block. Do not rely on this prompt alone for final release decisions; pair it with embedding-based semantic similarity checks and human spot-checking for the first several runs.

This prompt is designed to operate as a batch comparison engine, not a one-off chat interaction. You feed it two sets of answers—baseline and candidate—alongside the source documents, and it returns a structured claim-level verification report. The harness must enforce that every claim is explicitly grounded in a source passage before the hallucination rate is computed. Without this grounding step, the comparison degenerates into a fluency contest rather than a factual accuracy measurement.

Wire the prompt into a CI/CD evaluation step that runs after any prompt template change, retrieval pipeline update, or model version bump. The harness should: (1) load a golden QA dataset with known source documents and reference answers, (2) generate answers from both the baseline and candidate configurations, (3) call this comparison prompt with both answer sets and the source documents, (4) parse the structured output to extract hallucination counts, severity labels, and attribution gaps, and (5) compare the hallucination rate against a pre-defined threshold. If the candidate hallucination rate exceeds the baseline by more than a configurable margin (e.g., 5 percentage points), block the release and flag for human review.

Validation and retry logic is critical because the prompt produces structured JSON. Implement a JSON Schema validator that checks for required fields (claim_id, claim_text, verdict, evidence_citation, severity). If validation fails, retry once with the error message appended as a [CONSTRAINTS] update. If the retry also fails, log the raw output and escalate. For high-stakes domains like healthcare or legal QA, require human review of any claim marked unsupported or contradicted before the report is accepted. Log every comparison run with the prompt version, model identifier, dataset snapshot, and hallucination rate delta for auditability.

Model choice matters. Use a model with strong instruction-following and long-context handling for this task—GPT-4, Claude 3.5 Sonnet, or equivalent. Smaller models often collapse the structured output or skip evidence citations when documents are long. If you must use a smaller model, split the comparison into batches of 5–10 QA pairs and aggregate results. Never run this prompt without source documents; if your QA dataset lacks ground-truth passages, use a retrieval step to fetch candidate documents first, but flag that the grounding quality depends on retrieval recall.

What to avoid: Do not treat the hallucination rate as a single scalar without examining severity. A 2% hallucination rate where all errors are minor date offsets is different from a 2% rate where claims fabricate drug interactions. Always review the severity distribution before making a promotion decision. Do not skip the evidence-attribution step—this prompt's value is in the claim-to-source mapping, not just the final count. If you only need a binary pass/fail, use a simpler LLM judge; this prompt is for teams that need to understand what changed and why before they ship.