Retrieved Context vs Generated Output Diff Prompt

This prompt is designed for quality engineers and RAG developers who need to move beyond a binary 'hallucination' flag and into a detailed, actionable audit of answer faithfulness. The core job-to-be-done is to produce a structured, segment-level diff between a generated answer and the exact retrieved context that was provided to the model. You should use this when you have a production trace containing both the retrieved_context and the generated_output and you need a reviewable alignment report before shipping a model update, tuning a retrieval pipeline, or responding to a user-reported factual error. The output is not a pass/fail score; it is a forensic artifact that identifies added, omitted, distorted, and contradicted information, enabling precise root-cause analysis.

The ideal user is someone who already has access to a production trace and can extract the relevant text blocks. This prompt assumes the retrieved context is pre-extracted and formatted as a single input block. It does not perform retrieval itself, nor does it evaluate the quality of the retrieval pipeline. It is a targeted diagnostic tool for the generation step. For example, if a user reports that an answer misrepresented a contract clause, you would pull the trace, isolate the clause from the retrieved_context and the claim from the generated_output, and run this prompt to get a structured breakdown of exactly how the output diverged from the source material.

Do not use this prompt as a replacement for a full RAG pipeline evaluation or as a real-time guardrail. It is an offline analysis tool. It is not designed to catch every subtle form of bias or to judge the overall quality of the prose. Its strength is in creating a rigorous, evidence-based map of factual alignment and misalignment. After running this prompt, the next step is to use the structured diff to inform a specific action: fixing a prompt instruction, tuning a retrieval hyperparameter, adding a source to the knowledge base, or documenting a known model behavior. Avoid using the output to simply 'score' a model without acting on the specific segments it flags.

This prompt template is designed to be pasted directly into your trace analysis harness. It instructs the model to perform a structured comparison between the retrieved context and the generated output from a single production trace, producing a faithful diff. The goal is to isolate where the model's answer deviated from the provided evidence, which is the core diagnostic signal for debugging hallucination and grounding failures in RAG systems.

textCopy
You are a trace analysis auditor. Your task is to compare the RETRIEVED CONTEXT against the GENERATED OUTPUT from a single production trace and produce a structured diff.

## INPUTS

RETRIEVED CONTEXT:
[RETRIEVED_CONTEXT]

GENERATED OUTPUT:
[GENERATED_OUTPUT]

## OUTPUT SCHEMA

Return a valid JSON object with the following structure:
{
  "faithfulness_score": <float between 0.0 and 1.0, where 1.0 means the output is fully supported by the context>,
  "summary": "<one-sentence summary of the overall faithfulness>",
  "segments": [
    {
      "segment_id": "<unique identifier for this segment>",
      "generated_text": "<the exact text from the generated output being evaluated>",
      "alignment_status": "<FULLY_SUPPORTED | PARTIALLY_SUPPORTED | UNSUPPORTED | CONTRADICTED>",
      "supporting_context": "<the exact text from the retrieved context that supports or contradicts this segment, or null if none>",
      "issue_type": "<ADDED | OMITTED | DISTORTED | CONTRADICTED | NONE>",
      "explanation": "<brief explanation of the alignment or misalignment>"
    }
  ]
}

## CONSTRAINTS

- Compare the generated output against ONLY the provided retrieved context. Do not use external knowledge.
- If the generated output contains information not present in the context, mark it as ADDED.
- If the context contains important information missing from the output, add a segment with generated_text set to null and issue_type set to OMITTED.
- If the output distorts or misrepresents information from the context, mark it as DISTORTED.
- If the output directly contradicts the context, mark it as CONTRADICTED.
- Segment the generated output into atomic claims or sentences for granular analysis.
- The faithfulness_score should reflect the proportion of fully supported segments relative to total segments.

To adapt this template, replace [RETRIEVED_CONTEXT] with the raw text of the documents, passages, or chunks retrieved during the trace. Replace [GENERATED_OUTPUT] with the final answer the model produced. For high-risk domains such as healthcare or finance, always route the output of this prompt to a human reviewer before taking any action. You can also extend the output schema to include trace event IDs or document pointers if your observability platform captures them, enabling direct navigation from the diff back to the source trace.

This prompt is designed to run as a post-hoc analysis step within an observability or evaluation pipeline, not as a real-time user-facing feature. It expects a fully hydrated trace record containing both the retrieved context and the generated output, typically sourced from a tracing platform like LangSmith, Arize, or a custom trace store. The primary integration point is a batch or streaming job that pulls traces flagged for hallucination review—either by automated evals, user reports, or sampling rules—and enriches them with a structured diff and faithfulness score before routing to a human review queue or a dashboard.

Wire the prompt into an application by first constructing a trace retrieval function that fetches the required fields: [RETRIEVED_CONTEXT] (the full list of retrieved documents or passages with metadata), [GENERATED_OUTPUT] (the final model response), and optionally [USER_QUERY] for additional context. Validate these inputs before calling the model—reject traces where the retrieved context is empty, the generated output is truncated, or the trace event IDs are missing. On model response, parse the JSON output and validate that the faithfulness_score is a float between 0 and 1, that each segment in the diff has a non-empty type field matching the allowed enum (added, omitted, distorted, contradicted, faithful), and that trace_event_references contain valid span IDs from the original trace. If validation fails, retry once with the error message appended to the prompt as a [PREVIOUS_ERROR] constraint. For high-stakes domains like healthcare or finance, route all diffs with a faithfulness score below 0.9 or any contradicted segment to a human review queue before the finding is considered confirmed.

Model choice matters here: use a model with strong instruction-following and structured output capabilities, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 2.0 Flash, and set temperature=0 to minimize variance in the faithfulness score. Log every diff output alongside the original trace ID, model version, and prompt version to enable longitudinal tracking of hallucination rates. Avoid running this prompt on traces where the retrieved context exceeds the model's context window—chunk the context or use a summarization pre-processing step instead. The diff output should be treated as a diagnostic signal, not a final verdict; always correlate it with user feedback and downstream eval results before triggering automated rollbacks or model switches.