Multi-Turn Hallucination Accumulation Audit Prompt

This prompt is designed for RAG chat teams and evaluation engineers who need to detect and measure factual drift that compounds across conversation turns. Unlike single-turn hallucination checks, this audit tracks whether an unsupported claim made in turn three reappears or mutates in turn seven, creating a cascade of misinformation. The primary job-to-be-done is producing a turn-by-turn hallucination log that links each unsupported claim to its source evidence (or lack thereof) and traces its propagation into later responses. The ideal user is an ML engineer or evaluation lead integrating this into a CI/CD pipeline for a customer-facing chat product where groundedness is a non-negotiable safety property.

Required inputs include the full conversation transcript with turn indices, the source evidence chunk provided to the model for each assistant turn, and a groundedness threshold for flagging. The prompt expects structured evidence mapping—if your retrieval system doesn't log which chunks were passed to the model per turn, you cannot use this prompt effectively. Do not use this prompt for single-turn Q&A evaluation, for conversations without source evidence, or for measuring stylistic quality. It is specifically scoped to factual consistency against provided sources across time. For conversations where the model is expected to use general world knowledge rather than retrieved evidence, use a factuality evaluation prompt instead.

Production constraints matter here. This prompt is token-heavy because it processes full multi-turn transcripts with per-turn evidence. Expect higher latency and cost than single-turn evals. For long conversations exceeding 20 turns, consider windowing the audit into overlapping segments or using a sliding context window to avoid context-limit truncation. The output is a structured JSON log designed for automated ingestion into evaluation dashboards, not for human reading. Wire this into a regression test suite where every prompt or retrieval change triggers a re-audit of golden conversation datasets. If the audit flags a propagated hallucination, the failure mode is rarely the prompt alone—it's often a retrieval gap where the correct evidence wasn't provided to the model in the first place. Use the audit output to drive retrieval improvements, not just prompt fixes.

This prompt template is designed to be copied directly into your evaluation harness. It instructs the model to act as a strict, evidence-only auditor. The model receives the full conversation transcript and the source evidence provided for each assistant turn. Its job is to identify every factual claim in each assistant response, verify it against the corresponding source evidence, and track whether unsupported claims from earlier turns are repeated or built upon later. The output is a structured, turn-by-turn log that quantifies cumulative groundedness decay.

textCopy
You are a strict factual auditor. Your only task is to compare assistant responses against provided source evidence. You must never use your own knowledge. If evidence is missing, insufficient, or ambiguous, mark the claim as UNSUPPORTED.

## INPUT
[CONVERSATION_TRANSCRIPT]

## SOURCE EVIDENCE PER TURN
[SOURCE_EVIDENCE]

## AUDIT INSTRUCTIONS
For each assistant turn in the conversation:
1. Extract every discrete factual claim made by the assistant.
2. For each claim, locate the exact supporting passage in the source evidence provided for that specific turn.
3. If a claim cannot be matched to source evidence for that turn, check if it was introduced in a prior turn. If so, flag it as a PROPAGATED_HALLUCINATION and link to the turn where it first appeared.
4. Assign a grounding status to each claim: GROUNDED, UNSUPPORTED, PROPAGATED_HALLUCINATION, or CONTRADICTS_EVIDENCE.
5. Calculate a turn-level Groundedness Score: (number of GROUNDED claims) / (total claims in the turn).

## OUTPUT_SCHEMA
Return a single JSON object with this exact structure:
{
  "session_id": "string",
  "audit_turns": [
    {
      "turn_index": integer,
      "assistant_text": "string",
      "claims": [
        {
          "claim_text": "string",
          "status": "GROUNDED | UNSUPPORTED | PROPAGATED_HALLUCINATION | CONTRADICTS_EVIDENCE",
          "source_quote": "string or null",
          "propagated_from_turn": integer or null,
          "auditor_notes": "string"
        }
      ],
      "groundedness_score": float,
      "turn_notes": "string"
    }
  ],
  "cumulative_groundedness_score": float,
  "propagation_graph": [
    {
      "origin_turn": integer,
      "unsupported_claim": "string",
      "propagated_to_turns": [integer]
    }
  ],
  "critical_findings": ["string"]
}

## CONSTRAINTS
- Do not invent facts. If evidence is missing, mark the claim UNSUPPORTED.
- Do not use your own knowledge to fill gaps.
- A claim is a single, verifiable statement of fact. Opinions, pleasantries, and stylistic choices are not claims.
- If the assistant correctly refuses to answer due to lack of evidence, that is not a hallucination.
- The cumulative_groundedness_score is the mean of all turn-level groundedness scores.

To adapt this template, replace the [CONVERSATION_TRANSCRIPT] and [SOURCE_EVIDENCE] placeholders with your actual data. The source evidence structure should be a mapping of turn indices to the retrieved passages or documents available when that turn's response was generated. If your RAG system uses a single retrieval step for the entire session, note that in your harness documentation—the audit will still work, but the propagated_from_turn detection becomes more critical. For high-stakes domains like healthcare or finance, add a [RISK_LEVEL] placeholder that gates whether critical_findings must be reviewed by a human before the audit is considered complete. The output schema is intentionally strict; validate the JSON structure in your application layer before storing results, and log any parse failures for prompt refinement.

This prompt is designed to operate as a batch evaluation job, not a real-time chat interceptor. You feed it a complete conversation transcript and the corresponding source evidence for each assistant turn, and it returns a structured audit log. The primary integration point is your existing evaluation runner (e.g., a CI/CD pipeline, a nightly eval job, or a manual QA tool). The harness must enforce that source evidence is provided per turn; without it, the prompt cannot distinguish between a well-grounded claim and a fluent hallucination.

Input assembly is the most critical harness step. Before calling the model, construct a JSON object that pairs each assistant turn index with its [TURN_EVIDENCE]—the exact retrieved chunks, documents, or API responses available when that turn was generated. If your RAG system logs retrieval results per request, join them on turn_id. If not, you must replay retrievals or extract source metadata from the original generation logs. The harness should validate that every assistant turn in [CONVERSATION_TRANSCRIPT] has a corresponding evidence entry. Missing evidence for a turn should either skip that turn's audit or mark it as unevaluable rather than silently passing it.

Output validation must happen before results are trusted. The prompt returns a JSON object with a turn_audits array. Your harness should validate: (1) every turn index in the transcript appears exactly once, (2) each propagated_claims entry references a valid source turn index that is earlier than the current turn, (3) cumulative_groundedness_score values fall within the declared range, and (4) unsupported_claims arrays contain non-empty evidence_gap descriptions when a claim is flagged. Schema validation alone is insufficient—you need semantic checks. If validation fails, retry once with the validation errors injected into [CONSTRAINTS]. If it fails again, log the raw output and alert a human reviewer.

Model choice matters for this task. The prompt requires cross-turn reasoning, claim tracing, and structured output generation. Use a model with strong instruction-following and long-context handling (e.g., Claude 3.5 Sonnet, GPT-4o). Avoid smaller or older models that struggle with multi-hop reasoning across long transcripts. Set temperature to 0 or near-zero to minimize variance in audit results. If your conversation exceeds the model's context window, split it into overlapping windows of N turns and run the audit per window, then merge results with a deduplication step for claims that span window boundaries.

Logging and observability are essential because this prompt produces evidence used to gate releases or flag regressions. Log the full prompt payload (transcript + evidence), the raw model response, the validated output, and any validation errors. Store these alongside the conversation's session_id and the prompt version. This audit trail lets you debug disagreements between the LLM judge and human reviewers, and it provides the evidence needed to improve the prompt over time. If you're running this in a CI/CD pipeline, treat audit failures as blocking regressions only after confirming they are not false positives from ambiguous source material. Start with a shadow mode deployment where audits run without blocking, then promote to a gate after calibrating against human judgments on at least 50 conversations.