Hallucinated Source Detection Prompt

This prompt is a post-generation audit tool for compliance-sensitive AI applications that produce cited answers. Its primary job is to verify that every citation, reference, and source attribution in a generated text can be traced back to a provided retrieval corpus. The ideal user is an AI engineer or platform builder integrating this check into a RAG pipeline, a compliance officer validating outputs before they reach end users, or a QA lead building automated evaluation harnesses. You need two things to use this prompt: the generated text containing citations, and the complete set of source documents or passage IDs that were available during generation. Without both, the verification cannot proceed.

Use this prompt when fabricated citations create regulatory risk, when retrieval failures in your RAG pipeline could produce plausible-sounding but non-existent references, and when user trust depends on verifiable source provenance. It belongs in workflows where a single hallucinated citation can cause a compliance violation, a customer escalation, or a loss of credibility. The prompt returns a structured verdict for each citation—verified, hallucinated, or ambiguous—along with supporting evidence and a recommended action. This output can be wired directly into an automated gate that blocks responses with hallucinated sources or routes them for human review.

Do not use this prompt as a generation step. It is not designed to produce answers or fix citations; it only audits them. Do not use it when your application does not have a defined retrieval corpus to cross-reference against—the prompt cannot detect hallucination without ground truth evidence. Do not use it for real-time streaming responses where citations are still being formed. And do not treat it as a replacement for retrieval quality improvements; if your retriever consistently fails to surface relevant documents, this prompt will correctly flag the resulting citations as unsupported, but the root cause is retrieval failure, not citation fabrication. Pair this prompt with retrieval evaluation and retry logic before relying on it as your sole hallucination guard.

This prompt template is designed to be dropped directly into your validation harness. It instructs the model to act as a provenance auditor, cross-referencing every citation in a generated text against a provided list of verified source identifiers. The core job is to flag any reference that cannot be traced back to the retrieval corpus, distinguishing between a true hallucination and a minor formatting error. Before integrating, ensure your pipeline can supply the [VERIFIED_SOURCE_IDS] from your document store or vector database metadata.

codeCopy
You are a strict provenance auditor. Your task is to analyze the provided [GENERATED_TEXT] and identify any cited sources that are fabricated, non-existent, or cannot be matched to the [VERIFIED_SOURCE_IDS] list.

# INPUTS
- [GENERATED_TEXT]: The AI-generated answer containing inline citations.
- [VERIFIED_SOURCE_IDS]: A JSON array of unique identifiers for every document in the retrieval corpus. Example: ["doc_123", "doc_456.pdf", "ref_789"]
- [CITATION_FORMAT]: A description of the citation format used in the text (e.g., "[doc_id]", "(Author, Year)", "footnote number").

# TASK
1. Extract every distinct citation from [GENERATED_TEXT] based on [CITATION_FORMAT].
2. For each extracted citation, attempt to match it to an entry in [VERIFIED_SOURCE_IDS]. Use fuzzy matching for minor formatting differences (e.g., "doc_123" vs "DOC-123") but do not guess.
3. Classify each citation into one of three categories:
   - **VERIFIED**: The citation matches a known source ID.
   - **HALLUCINATED**: The citation does not match any known source ID and appears fabricated.
   - **FORMAT_ERROR**: The citation is a close match to a known source ID but has a typo or formatting inconsistency.
4. For every HALLUCINATED or FORMAT_ERROR citation, quote the specific sentence in [GENERATED_TEXT] where it appears.

# OUTPUT_SCHEMA
Return a single JSON object with the following structure:
{
  "audit_result": {
    "total_citations_found": <integer>,
    "verified_count": <integer>,
    "hallucinated_count": <integer>,
    "format_error_count": <integer>,
    "flagged_citations": [
      {
        "extracted_citation_text": "<string>",
        "status": "HALLUCINATED" | "FORMAT_ERROR",
        "context_sentence": "<string>",
        "suggested_correction": "<string or null>"
      }
    ]
  }
}

# CONSTRAINTS
- Do not invent source IDs. Only match against [VERIFIED_SOURCE_IDS].
- If a citation is ambiguous, flag it as HALLUCINATED.
- Output only the JSON object. No markdown fences or explanatory text.

To adapt this template, replace the placeholders with your application's specific data structures. The [VERIFIED_SOURCE_IDS] list should be dynamically generated from your retrieval step's metadata. For high-stakes compliance applications, the output of this prompt should not be the final step. Pipe the flagged_citations array into a human review queue or a secondary verification tool that can perform a live lookup against the source database. This prompt is a detection mechanism, not a correction mechanism; pair it with a 'Citation Insertion Repair' or 'Answer Regeneration' prompt to close the loop.

This prompt is designed to sit inside a post-generation validation harness, not as a standalone chat interaction. After your primary system produces a cited answer, pass both the generated text and the full retrieval context into this prompt. The harness should treat the prompt's output as a structured verdict—not a suggestion—and use it to decide whether the answer can be served to the user, requires regeneration, or must be escalated for human review.

Wire the prompt into a validation loop with a configurable retry budget. On a verdict: FAIL, extract the list of hallucinated sources and feed them back into your answer-generation prompt as negative constraints—for example, 'Do not cite any of the following fabricated sources: [LIST].' Retry up to [MAX_RETRIES] times, typically 2–3. After the budget is exhausted, route the answer to a human review queue with the hallucination report attached. Log every invocation: the input context hash, the model and version used, the verdict, the list of flagged sources, and the retry count. This trace is essential for debugging retrieval gaps and for audit evidence in compliance-sensitive domains.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities—GPT-4o, Claude 3.5 Sonnet, or equivalent. Avoid smaller or older models that may conflate plausible-sounding source names with actual retrieval results. If your retrieval corpus is large, consider truncating the provided context to the top-N passages actually used in the answer to keep the prompt within token limits. Validate the output schema strictly: if the JSON is malformed or missing required fields, treat it as a FAIL and increment the retry counter. Never serve an answer that hasn't passed this check to a user in a compliance-sensitive workflow.

For high-stakes applications, add a human-in-the-loop gate after the first FAIL verdict. Present the reviewer with the original answer, the flagged sources, and the retrieval context side by side. The reviewer can confirm the hallucination, override a false positive, or mark the source as legitimate but poorly formatted. Feed these human decisions back into your eval dataset to improve the prompt's precision over time. Avoid using this prompt as the sole defense in regulated domains—pair it with retrieval-audit logging, citation-mandatory generation prompts, and periodic manual audits of served answers.