Factual Drift Detection in Multi-Turn RAG Traces Prompt

This prompt is built for a specific production diagnosis: a conversational AI system using Retrieval-Augmented Generation (RAG) has produced a multi-turn session that looks correct turn-by-turn but may contain cross-turn factual contradictions. The job-to-be-done is a structured, trace-anchored audit that compares atomic factual claims across every turn, checks each claim against the retrieved context captured in the trace for that specific turn, and identifies three failure modes: contradictions with earlier grounded statements, new claims introduced without support in the current turn's retrieved context, and factual qualifications present in an earlier turn that were dropped in a later turn. The ideal user is an AI reliability engineer, a conversational AI developer, or a quality engineer who has access to a full trace export containing user messages, model responses, and per-turn retrieval payloads. Without per-turn retrieval data, the prompt cannot perform its core function of distinguishing a retrieval gap from a generation error.

Do not use this prompt for single-turn evaluations. A single turn has no cross-turn drift to detect, and a dedicated faithfulness or citation verification prompt will give you a more focused analysis with less token overhead. Do not use it for real-time guardrailing during generation; this prompt is designed for offline trace analysis and expects a complete session payload, not a streaming partial response. Do not use it when the trace does not include per-turn retrieval data—if you only have final answers without the context the model saw at each step, you cannot determine whether a new claim was unsupported by retrieval or was a contradiction of a prior grounded statement. For those cases, use a standalone hallucination detection prompt that operates on a single (context, output) pair. Also avoid this prompt when the session involves heavy tool-call sequences where factual claims are derived from tool outputs rather than retrieved passages; the prompt's grounding logic assumes retrieved context as the evidence source, and tool-call traces require a different audit structure.

Before running this prompt, verify that your trace format includes turn boundaries, user messages, model responses, and the retrieved context payload for each turn. If your tracing system captures these fields under different keys, adapt the [TRACE_JSON] placeholder schema before use. The prompt works best on sessions with three or more turns, where drift patterns become detectable. For sessions with only two turns, the analysis will still run but may produce a sparse drift timeline. After running the prompt, expect a structured drift timeline that references specific turn numbers and trace spans, which you can use to file bug reports, update retrieval pipelines, or adjust system instructions. If the output flags a contradiction, always verify it manually against the trace before treating it as a confirmed defect—the prompt is a diagnostic aid, not a final arbiter.

This prompt template is designed to be dropped directly into your trace analysis pipeline. It instructs the model to extract atomic claims from each assistant turn, ground them against the retrieved context for that turn, and then compare claims across turns to identify contradictions, unsupported new facts, and qualifications that were silently dropped. The output is a structured drift timeline that references specific trace events, making it auditable and actionable for debugging conversational RAG systems.

textCopy
You are an expert trace auditor analyzing a multi-turn RAG conversation for factual drift.

Your task is to:
1. Extract every atomic factual claim from each assistant response in the trace.
2. For each claim, determine whether it is supported by the retrieved context for that turn. Mark it as GROUNDED, PARTIALLY_GROUNDED, or UNSUPPORTED.
3. Compare claims across all turns to detect:
   - Contradictions: A later claim that directly conflicts with an earlier grounded claim.
   - Unsupported new claims: A claim introduced in a later turn that has no support in that turn's retrieved context.
   - Dropped qualifications: An earlier claim that included a caveat, confidence qualifier, or limitation that was omitted in a later restatement.
4. Produce a drift timeline ordered by turn, with each event referencing the specific trace event IDs.

## INPUT

Trace JSON:
[TRACE_JSON]

## OUTPUT SCHEMA

Return a JSON object with this exact structure:
{
  "trace_id": "string",
  "total_turns": integer,
  "drift_events": [
    {
      "event_id": "string",
      "turn_number": integer,
      "drift_type": "CONTRADICTION" | "UNSUPPORTED_NEW_CLAIM" | "DROPPED_QUALIFICATION",
      "earlier_claim": {
        "turn": integer,
        "claim_text": "string",
        "trace_event_ref": "string",
        "grounding_status": "GROUNDED" | "PARTIALLY_GROUNDED" | "UNSUPPORTED"
      },
      "later_claim": {
        "turn": integer,
        "claim_text": "string",
        "trace_event_ref": "string",
        "grounding_status": "GROUNDED" | "PARTIALLY_GROUNDED" | "UNSUPPORTED"
      },
      "description": "string explaining the drift",
      "severity": "CRITICAL" | "MAJOR" | "MINOR"
    }
  ],
  "summary": {
    "total_drift_events": integer,
    "by_type": {
      "CONTRADICTION": integer,
      "UNSUPPORTED_NEW_CLAIM": integer,
      "DROPPED_QUALIFICATION": integer
    },
    "overall_assessment": "string"
  }
}

## CONSTRAINTS

- Only flag a claim as UNSUPPORTED if the retrieved context for that specific turn does not contain evidence for it. Do not use context from other turns.
- A contradiction requires that the earlier claim was GROUNDED or PARTIALLY_GROUNDED. If both claims are unsupported, flag each as UNSUPPORTED_NEW_CLAIM separately rather than as a contradiction.
- For DROPPED_QUALIFICATION, the earlier claim must include an explicit qualifier (e.g., "likely", "estimates suggest", "as of Q2") that is absent in the later restatement.
- Severity: CRITICAL for safety, legal, or financial claims. MAJOR for factual errors with user impact. MINOR for imprecise but not misleading changes.
- If no drift is detected, return an empty drift_events array and an overall_assessment indicating the trace is consistent.

To adapt this template, replace [TRACE_JSON] with your actual trace data. The trace should include, at minimum, the user query, retrieved context, and assistant response for each turn, along with unique event IDs. If your trace format differs, adjust the extraction logic in the prompt's instructions accordingly. For high-stakes domains such as healthcare or finance, always route CRITICAL severity findings to human review before taking action. Pair this prompt with a JSON schema validator in your pipeline to catch malformed outputs before they enter your drift monitoring dashboard.

This prompt is designed to be integrated into a post-hoc trace analysis pipeline, not a real-time chat flow. It expects a fully hydrated multi-turn trace containing user messages, assistant responses, and the retrieved context for each turn. The primary integration point is a batch job or a manual review tool that queries your trace store, assembles the required inputs, and invokes the LLM with this prompt. The output is a structured drift timeline that should be written back to your observability platform as annotations on the original trace, enabling search and alerting on drift_severity and contradiction_type fields.

To wire this into an application, build a thin service that performs the following steps: (1) Query your trace database for sessions exceeding a turn threshold or flagged by user reports. (2) For each session, extract the [CONVERSATION_HISTORY] and [RETRIEVED_CONTEXT_PER_TURN] arrays, ensuring each turn object includes a unique turn_id and timestamp. (3) Populate the prompt template, setting [DRIFT_CATEGORIES] to your organization's taxonomy (e.g., 'direct contradiction', 'unsupported new fact', 'tense or entity shift'). (4) Call a capable model (GPT-4o or Claude 3.5 Sonnet) with response_format set to a JSON schema matching the expected [OUTPUT_SCHEMA]. (5) Validate the response: confirm every drift_event references a valid turn_id from the input, that source_claim and contradicting_claim are non-empty strings, and that severity is within your enum. On validation failure, retry once with the validation error injected into the [CONSTRAINTS] field. Log all validation failures and retries for prompt debugging.

For high-stakes domains such as healthcare or legal applications, do not automate downstream actions based solely on the drift detection output. Instead, route traces with severity: critical to a human review queue. Store the full prompt payload and model response as an audit record. Implement a confidence threshold: if the model's self-reported detection_confidence is below 0.7, suppress automated alerts and flag for manual triage. Avoid wiring this prompt directly into a model response interceptor that blocks or rewrites the assistant's output in real time; the latency and cost profile of multi-turn analysis is better suited for asynchronous monitoring. Start with a daily batch over high-risk sessions before moving to near-real-time streaming.