Evidence Grounding Trace Audit Prompt

This prompt is a precision diagnostic tool for RAG developers and quality engineers who need to audit a single production trace for evidence faithfulness. It maps every factual claim in the model's final output to specific retrieved passages or source documents, flags unsupported statements, and identifies citation errors. Use this when a trace shows a suspicious answer, a user reports a hallucination, or you are running a spot check on grounding quality. This prompt assumes you already have a reconstructed trace containing the user query, retrieved context chunks with source identifiers, and the final model output.

This prompt does not replace automated evaluation pipelines; it is a deep-inspection instrument for one session at a time. Do not use it for bulk hallucination scoring across thousands of traces—it is too expensive and verbose for that. Do not use it when the trace is missing retrieval spans or source identifiers, because the audit cannot ground claims without evidence. Do not use it to judge answer quality, tone, or completeness; its only job is to verify whether factual statements are supported by the retrieved context. If you need aggregate metrics, pair this with an LLM judge or evaluation harness that samples traces and applies this prompt as a spot-check step.

Before running this prompt, ensure your trace includes the user query, a list of retrieved context chunks with unique source IDs, and the final model output. If your RAG pipeline re-ranks or summarizes chunks before generation, include the final context window contents, not the raw retrieval results. The prompt will produce a claim-by-claim grounding report. After receiving the output, review any unsupported claims manually, cross-check flagged citation errors against the original sources, and decide whether the trace indicates a retrieval gap, a generation failure, or a citation formatting bug. Use the findings to tighten retrieval, adjust system instructions, or add citation post-processing.

This prompt template is designed to be pasted directly into your LLM interface or trace review tool. It instructs the model to act as a forensic auditor, comparing the final output of a RAG pipeline against the source documents that were present in the context window during that specific request. The goal is to produce a structured, claim-by-claim grounding report that flags unsupported statements, citation errors, and evidence gaps.

textCopy
You are an evidence-grounding auditor. Your task is to analyze a single production trace from a Retrieval-Augmented Generation (RAG) system. You will be given the final model output and the exact set of retrieved source documents that were in the context window.

Your job is to produce a strict, claim-by-claim grounding report. Follow these rules:
1. Decompose the final model output into a list of discrete, verifiable factual claims.
2. For each claim, determine if it is fully supported, partially supported, or unsupported by the provided source documents.
3. For supported claims, you MUST provide the exact text span from the source documents that serves as evidence. Include the source document ID.
4. If a claim is unsupported, flag it as a potential hallucination.
5. If the output contains a citation that points to a source document which does not support the associated claim, flag it as a citation error.
6. Do not use your own world knowledge. Base your judgment ONLY on the provided source documents.

## INPUT DATA

<final_model_output>
[FINAL_OUTPUT]
</final_model_output>

<retrieved_source_documents>
[RETRIEVED_DOCUMENTS]
</retrieved_source_documents>

## OUTPUT FORMAT

Produce a JSON object with the following structure. Do not include any text outside the JSON object.

{
  "audit_summary": {
    "total_claims": <integer>,
    "fully_supported_claims": <integer>,
    "partially_supported_claims": <integer>,
    "unsupported_claims": <integer>,
    "citation_errors": <integer>,
    "overall_grounding_score": <float between 0.0 and 1.0>
  },
  "claims": [
    {
      "claim_id": <integer>,
      "claim_text": "<string>",
      "verdict": "fully_supported | partially_supported | unsupported",
      "evidence": [
        {
          "source_document_id": "<string>",
          "evidence_text": "<exact string from source>"
        }
      ],
      "citation_error": <boolean>,
      "notes": "<string, optional explanation>"
    }
  ]
}

To adapt this template, replace the [FINAL_OUTPUT] and [RETRIEVED_DOCUMENTS] placeholders with data extracted from your trace observability platform. The [RETRIEVED_DOCUMENTS] should be a structured list, where each document has a unique ID and its full text content. For high-stakes domains like healthcare or finance, the notes field in the output is critical for human reviewers to understand the auditor's reasoning. Always run this prompt with a low temperature setting (e.g., 0.0-0.2) to maximize deterministic, repeatable evaluations.

The Evidence Grounding Trace Audit Prompt is designed to be called programmatically as a post-hoc analysis step, not as a real-time user-facing feature. In a production harness, you should trigger this prompt after a user session completes and the full trace—including the user query, retrieved passages, final output, and any intermediate tool calls—has been persisted to your observability store. The prompt expects a structured input payload containing the final generated text and an array of source documents with their identifiers and content. Wire this prompt into an asynchronous job queue (e.g., a background worker or serverless function) that listens for trace-completion events from your inference pipeline. This keeps the audit out of the critical path for user latency while ensuring every session is eventually reviewed.

Before calling the model, validate the input payload against a strict schema. The [OUTPUT_TEXT] field must be a non-empty string, and the [SOURCE_DOCUMENTS] array must contain at least one object with id and content fields. If the trace is incomplete—for example, if the retrieval step failed and returned zero documents—skip the audit and log a trace_incomplete event instead of sending a malformed request. After receiving the model's response, parse the JSON output and validate that every claim object contains the required fields: claim_text, grounding_status (one of SUPPORTED, PARTIALLY_SUPPORTED, UNSUPPORTED), and source_references (an array of source IDs). If the model returns malformed JSON, use a repair prompt with the original schema as context and retry once. If the second attempt also fails, escalate the trace for human review and log the failure for schema drift monitoring.

For high-stakes domains such as healthcare, legal, or finance, do not treat this prompt's output as a final verdict. Route any trace where the UNSUPPORTED claim count exceeds a configurable threshold (e.g., more than 10% of total claims) to a human review queue. Additionally, implement a calibration harness that periodically samples audited traces and has a domain expert label a subset of claims independently. Compare the human labels against the model's grounding_status classifications to measure precision and recall over time. This feedback loop is essential because grounding evaluation is subjective at the margins, and model performance can drift as retrieval pipelines or underlying documents change. Log every audit result—including the trace ID, claim counts, grounding scores, and any human overrides—to your observability platform so you can track grounding quality trends across prompt versions and document corpus updates.