Factual Drift Detection in Long-Form Output

This playbook is for engineering teams who generate long-form content—reports, articles, summaries, or documentation—from source material and need to catch where the output gradually diverges from the evidence. Unlike single-claim hallucination checks, factual drift detection identifies the specific sentence where grounding weakens and measures how the severity accumulates across the output. The ideal user is an AI engineer or evaluation lead building automated quality gates for a production pipeline where outputs routinely exceed 500 words and source fidelity must be auditable at scale.

Use this prompt when you have a complete generated text and the full source material it was supposed to be grounded in. The prompt requires both pieces as input and expects a structured drift report: the first sentence where grounding weakens, a severity score per section, and a cumulative drift trajectory. Do not use this prompt for short-form outputs under 200 words, for real-time streaming generation where the full text isn't available, or when the source material is ambiguous or multi-document without clear provenance. The prompt assumes a single, authoritative source context; for multi-source RAG outputs with conflicting evidence, pair this with a source conflict detection prompt first.

This prompt is a detection tool, not a repair tool. It tells you where and how badly grounding failed, but it does not rewrite the output. After running drift detection, you should route flagged outputs to a repair or regeneration workflow, log the drift report for monitoring dashboards, and escalate high-severity drift to human review. For regulated or high-stakes domains—healthcare, legal, finance—always require human sign-off on drift reports before any downstream action. Wire this into your CI/CD evaluation suite as a pre-release gate, not as a one-off spot check.

This template is designed to detect factual drift—the gradual divergence of a generated text from its source material. Unlike simple hallucination checks that flag individual unsupported claims, this prompt identifies the sentence where grounding weakens and measures the cumulative severity of drift across the output. It is built for teams generating long-form content such as reports, articles, summaries, or documentation where a model may start faithfully but slowly introduce unsupported assertions, exaggerations, or fabrications as the output lengthens.

codeCopy
You are a factual drift auditor. Your task is to compare a [GENERATED_OUTPUT] against its [SOURCE_MATERIAL] and identify where the generated text begins to diverge from the source.

## INPUT
- Generated Output: [GENERATED_OUTPUT]
- Source Material: [SOURCE_MATERIAL]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "drift_detected": boolean,
  "drift_start_sentence_index": integer or null,
  "drift_start_sentence_text": string or null,
  "drift_severity": "none" | "minor" | "moderate" | "severe" | "critical",
  "sentence_by_sentence_analysis": [
    {
      "sentence_index": integer,
      "sentence_text": string,
      "grounding_status": "fully_grounded" | "partially_grounded" | "ungrounded" | "contradicted",
      "supporting_source_excerpt": string or null,
      "drift_explanation": string or null
    }
  ],
  "cumulative_drift_summary": string,
  "recommended_action": "pass" | "flag_for_review" | "reject" | "regenerate_from_drift_point"
}

## CONSTRAINTS
- [CONSTRAINTS]

## INSTRUCTIONS
1. Read the entire [SOURCE_MATERIAL] first to establish the ground truth.
2. Process the [GENERATED_OUTPUT] sentence by sentence.
3. For each sentence, determine whether its factual content is:
   - **fully_grounded**: All claims are directly supported by the source.
   - **partially_grounded**: Some claims are supported, but others are not present in the source.
   - **ungrounded**: Claims are not found in the source material at all.
   - **contradicted**: Claims directly conflict with the source material.
4. Identify the first sentence where grounding status shifts from fully_grounded to any weaker status. This is the drift start point.
5. Assess cumulative drift severity across the entire output, not just the first divergence.
6. If no drift is detected, set drift_start_sentence_index to null and drift_severity to "none".
7. Provide a cumulative_drift_summary that explains the pattern of divergence.
8. Recommend an action based on severity and the output's intended use context: [RISK_LEVEL].

## EXAMPLES
[EXAMPLES]

Adaptation guidance: Replace [GENERATED_OUTPUT] with the long-form text under evaluation and [SOURCE_MATERIAL] with the reference documents, retrieved context, or ground-truth data. Use [CONSTRAINTS] to inject domain-specific rules such as acceptable paraphrase boundaries, tolerance for inferred conclusions, or field-specific fact definitions. The [EXAMPLES] placeholder should be populated with 2–3 annotated few-shot examples showing both drift and no-drift cases, including edge cases where drift is subtle. Set [RISK_LEVEL] to low, medium, high, or critical to calibrate the recommended_action threshold—higher risk levels should bias toward flag_for_review or reject for moderate drift.

What to do next: Wire this prompt into a post-generation evaluation step in your pipeline. Run it after every long-form generation where source fidelity matters. Store the structured JSON output for monitoring dashboards and regression testing. If drift is detected at severe or critical levels, trigger a regeneration from the drift start point rather than attempting a full rewrite. For high-risk domains, always route flag_for_review outputs to a human reviewer before publication.

This harness treats the drift detection prompt as a scoring function inside a larger evaluation loop. The prompt expects a long-form generated text and its source material, and it returns a structured JSON report identifying the first sentence where grounding weakens and a cumulative drift severity score. In production, you should not call this prompt once and trust the result. Instead, wrap it in a validation layer that checks the output schema, retries on malformed responses, and routes high-severity drift cases to a human review queue. The harness must also handle chunking for outputs that exceed the model's context window, since long-form content can easily span tens of thousands of tokens before the source material is even added.

Schema validation and retry logic form the first layer of the harness. The prompt template requests a specific JSON output shape with fields like drift_start_sentence_index, drift_severity_score, and sentence_annotations. After each model call, validate that the response parses as valid JSON, contains all required fields, uses the correct types, and falls within expected ranges (e.g., severity scores between 0.0 and 1.0). If validation fails, retry up to two times with an error-repair prompt that includes the original output and a specific validation error message. After three failures, log the raw response and escalate for manual review rather than silently accepting a broken output. For high-stakes content—medical summaries, legal analysis, financial reports—consider requiring a second judge model call and comparing drift scores; if the two judges disagree by more than 0.2 on severity, flag for human adjudication.

Chunking and context management are essential for long-form outputs. If the generated text plus source material exceeds your model's context limit, split the generated text into overlapping chunks of 2,000–4,000 tokens with a 200-token overlap between chunks. Run the drift detection prompt on each chunk independently, always including the full source material as context. Then merge the results by scanning chunk annotations sequentially: the first chunk where drift is detected marks the overall drift start point, and the maximum severity score across chunks becomes the overall drift severity. Be aware that chunk boundaries can mask drift that spans across them, so the overlap is critical. Log chunk-level results alongside the merged report for debugging. For models with native long-context support (128K+ tokens), you may skip chunking but should still monitor latency and cost, as drift detection on very long texts can become expensive at scale.

Human review routing should be triggered by configurable thresholds. Set a severity threshold (e.g., 0.7 or above) that automatically creates a review task in your queue system, attaching the generated text, source material, drift report, and the specific sentences flagged as ungrounded. For regulated domains, lower this threshold to 0.4 and require human sign-off before the content reaches end users. Store every drift detection result—including passed checks—in an evaluation log with timestamps, model version, prompt version, and source document identifiers. This audit trail lets you track drift patterns over time, identify which content types or sources correlate with higher drift rates, and measure whether prompt or model changes improve grounding. Avoid the temptation to auto-correct drifted content using the same model that produced it; instead, route corrections through a separate repair prompt or human editor to prevent compounding errors.