Factual Drift Detection Prompt Across Multi-Turn Answers

This playbook is for conversational RAG builders who need to maintain factual consistency across a session. The prompt compares the factual content of the current generated answer against all previous answers in the conversation history. It detects contradictions, retractions, and factual drift that would confuse users or violate trust. Use this prompt as a post-generation guardrail before the current answer is delivered to the user. It is not a replacement for per-turn source grounding checks; it is a cross-turn consistency layer that catches errors single-turn verification misses.

Deploy this prompt when your application maintains a multi-turn conversation history and the factual integrity of that history is critical—common in customer support, legal research, clinical summarization, and financial analysis copilots. The ideal user is an AI engineer or QA pipeline builder who already has a working per-turn RAG verification step but observes that answers contradict earlier turns when the model loses context, reinterprets evidence, or hallucinates to fill gaps. This prompt is most effective when the conversation history is well-structured, with clear delineation between user queries and system answers, and when each answer has already passed a single-turn source grounding check.

Do not use this prompt as your only factual guardrail. It assumes that each individual answer is already grounded in its own retrieved evidence; it only checks for cross-turn consistency. It will not catch a hallucination that is consistent across turns but unsupported by any source. Avoid using this prompt on very long conversation histories without summarization or windowing, as the model's attention will degrade. For high-stakes domains like healthcare or legal, always route detected drift events to a human reviewer rather than auto-correcting. The next step after implementing this check is to build a drift resolution workflow that can either roll back the contradictory statement, surface the conflict to the user, or trigger a re-answer with explicit history constraints.

This template is designed to be pasted directly into your evaluation harness, guardrail step, or post-generation check. It compares the current assistant answer against the conversation history, specifically looking for factual contradictions, unsupported retractions, and statements that silently reverse prior answers. The prompt expects the full conversation transcript and the current answer, and it outputs a structured drift report that your application can parse to decide whether to flag, regenerate, or escalate.

codeCopy
SYSTEM INSTRUCTION:
You are a factual consistency auditor for a multi-turn conversational AI system. Your job is to compare the CURRENT ANSWER against all previous ASSISTANT ANSWERS in the conversation history. You must detect factual drift, defined as any of the following:

1. CONTRADICTION: The current answer states a fact that directly conflicts with a fact stated in a previous assistant answer.
2. RETRACTION: The current answer withdraws or reverses a previous factual claim without acknowledging the change.
3. UNSOURCED CORRECTION: The current answer changes a previous factual claim but provides no new source or evidence for the change.
4. SILENT UPDATE: The current answer adds new factual details that materially alter the meaning of a previous answer without signaling the update.

You must NOT flag:
- Clarifications that add detail without contradicting prior facts.
- Explicit corrections where the assistant acknowledges the prior error.
- Answers to different questions that naturally cover different facts.
- Stylistic rephrasing that preserves the same factual content.

INPUT:

CONVERSATION HISTORY:
[CONVERSATION_HISTORY]

CURRENT ANSWER:
[CURRENT_ANSWER]

SOURCE EVIDENCE FOR CURRENT ANSWER (if available):
[SOURCE_EVIDENCE]

OUTPUT_SCHEMA:
Return a JSON object with the following structure:
{
  "drift_detected": boolean,
  "drift_instances": [
    {
      "drift_type": "CONTRADICTION" | "RETRACTION" | "UNSOURCED_CORRECTION" | "SILENT_UPDATE",
      "current_answer_excerpt": "exact text from current answer",
      "previous_answer_excerpt": "exact text from prior assistant turn",
      "prior_turn_index": number,
      "explanation": "why this constitutes factual drift",
      "severity": "HIGH" | "MEDIUM" | "LOW"
    }
  ],
  "overall_assessment": "summary of consistency across turns",
  "recommended_action": "FLAG_FOR_REVIEW" | "REGENERATE" | "NO_ACTION" | "ESCALATE"
}

CONSTRAINTS:
[CONSTRAINTS]

EXAMPLES:
[EXAMPLES]

To adapt this template, replace the square-bracket placeholders with your actual data. [CONVERSATION_HISTORY] should contain the full transcript, with clear speaker labels (e.g., USER: and ASSISTANT:) and turn indices. [CURRENT_ANSWER] is the latest assistant response you want to audit. [SOURCE_EVIDENCE] is optional but strongly recommended; when provided, the model can distinguish between a legitimate correction grounded in new evidence and an unsupported factual reversal. [CONSTRAINTS] can include domain-specific rules, such as 'Do not flag changes in numerical precision from two decimal places to one unless the rounding changes the substantive claim.' [EXAMPLES] should include at least one positive example (drift detected) and one negative example (no drift) formatted in the same JSON output schema to calibrate the model's judgment.

After pasting this template into your guardrail step, validate the output against the schema before allowing it to influence the user-facing response. For high-stakes domains such as healthcare or finance, route any output where drift_detected is true and severity is HIGH to a human review queue. For lower-stakes applications, you can use the recommended_action field to automate regeneration with a correction preamble or to append a consistency note to the user. Always log drift instances with the prior turn index and excerpts to support debugging and prompt improvement over time.

Integrating the factual drift detection prompt into a conversational RAG system requires treating it as a post-generation guardrail, not a synchronous step in the answer pipeline. After the primary model generates a response for the current turn, the harness should assemble a structured payload containing the new answer, the conversation history (or a window of prior assistant answers), and any relevant source evidence. This payload is then sent to the drift detection prompt, which returns a structured verdict indicating whether contradictions, retractions, or unsupported factual shifts exist. The harness must parse this verdict and decide whether to block the answer, flag it for review, or allow it through with a warning annotation.

The implementation should include a validation layer that enforces the expected output schema from the drift detection prompt. If the model returns malformed JSON, missing required fields like drift_detected or contradicting_statements, or an unparseable response, the harness should retry with the same payload up to a configured limit (typically 2-3 attempts). Each retry should append the previous parse error to the prompt context so the model can self-correct. For high-stakes domains such as clinical or legal RAG, the harness should route any detected drift to a human review queue rather than automatically blocking or rewriting the answer. The review interface should display the current answer, the conflicting prior statement, and the source evidence side by side.

Model choice matters for this workflow. Drift detection requires strong comparative reasoning across multiple turns, so prefer models with robust instruction-following and long-context handling. Avoid routing this check to smaller, faster models unless you have calibrated their performance on your specific conversation patterns. Log every drift detection result—including false positives and false negatives—to build an evaluation dataset over time. This dataset becomes your regression suite for testing prompt changes and model upgrades. Finally, consider implementing a severity threshold: minor rephrasings that don't change factual content should not trigger a block, while outright contradictions on safety-critical facts should halt the response immediately and escalate.