Source Document Grounding Scorecard Prompt

This prompt is designed for offline trace analysis and automated evaluation workflows, not for real-time user-facing responses. Its primary job is to produce a structured, per-document grounding scorecard by comparing a final generated output against the full list of retrieved documents captured in a production trace. The ideal user is a RAG developer, quality engineer, or AI reliability engineer who needs to populate retrieval quality dashboards, debug hallucination root causes, or generate structured evidence for eval pipelines. It assumes you already have a production trace containing the generated output, the list of retrieved documents with their content, and trace event identifiers.

Use this prompt when you need to answer specific diagnostic questions: Which retrieved documents were actually used in the final answer? Were they used faithfully, or was their content distorted? Which documents were ignored entirely, and was that a correct decision? The output is a per-document scorecard classifying each source as Used Faithfully, Used Partially, Used Unfaithfully, or Ignored, with a brief justification anchored to the trace. This structured output is ideal for feeding into monitoring dashboards, triggering alerts when grounding fidelity drops, or building datasets for fine-tuning retrieval and generation models.

Do not use this prompt for real-time guardrails that block responses before they reach the user—its analysis is too detailed and latency-intensive for synchronous use. It is also not a replacement for human review in high-risk domains like healthcare or legal applications; the scorecard should inform human judgment, not automate it. For live user-facing systems, pair this offline analysis with a lightweight, real-time faithfulness check. If your trace does not contain the full text of retrieved documents, this prompt will produce unreliable results because it cannot verify grounding against source content it cannot see. Ensure your tracing infrastructure captures complete document payloads before running this analysis.

This prompt template is designed to be dropped directly into your RAG evaluation pipeline. It takes a production trace—containing the user query, the list of retrieved documents, and the final generated answer—and produces a per-document grounding scorecard. The scorecard classifies each document as used, partially used, or ignored, and then assesses whether its use was faithful (accurately represented) or unfaithful (distorted, hallucinated, or contradicted). The output is structured for direct ingestion into retrieval quality dashboards, allowing you to monitor the precision and recall of your retrieval step in terms of actual generation impact, not just vector similarity.

textCopy
You are an expert retrieval-augmented generation (RAG) auditor. Your task is to analyze a production trace and produce a per-document grounding scorecard.

## INPUT DATA

[USER_QUERY]
[RETRIEVED_DOCUMENTS]
[GENERATED_ANSWER]

## TASK

For each document in [RETRIEVED_DOCUMENTS], perform the following analysis:

1.  **Usage Classification**: Determine if the document's content was used in [GENERATED_ANSWER].
    *   `used`: Specific claims or information from this document appear in the answer.
    *   `partially_used`: Some information was used, but other parts were ignored or a key point was missed.
    *   `ignored`: No information from this document is present in the answer.

2.  **Faithfulness Assessment**: If the document was `used` or `partially_used`, evaluate if its information was represented faithfully.
    *   `faithful`: The information in the answer accurately reflects the source document without distortion.
    *   `unfaithful`: The answer hallucinates details not in the document, contradicts the document, or misinterprets its core claim.
    *   `n/a`: For documents classified as `ignored`.

3.  **Evidence**: Provide a short, verbatim quote from the document and the corresponding sentence from the answer to justify your classification. If `ignored`, provide a quote from the document that was most relevant to the query but missed.

## OUTPUT FORMAT

Return a single JSON object with a key `"scorecard"` containing an array of objects. Each object must conform to this schema:

{
  "document_id": "string",
  "usage": "used" | "partially_used" | "ignored",
  "faithfulness": "faithful" | "unfaithful" | "n/a",
  "evidence": {
    "source_quote": "string",
    "answer_quote": "string"
  },
  "notes": "string (optional, for brief auditor commentary)"
}

## CONSTRAINTS

*   Base your analysis strictly on the provided [RETRIEVED_DOCUMENTS] and [GENERATED_ANSWER]. Do not use outside knowledge.
*   If the answer contains information not found in any document, note this in the `notes` field of the most relevant document or flag it as an overall issue in a final `ungrounded_claims` array.
*   Be precise with quotes. The `source_quote` and `answer_quote` must be verbatim substrings.
*   If [RETRIEVED_DOCUMENTS] is empty, return an empty `scorecard` array and a note.

To adapt this template, replace the square-bracket placeholders with data from your trace logger. [USER_QUERY] is the raw user input. [RETRIEVED_DOCUMENTS] should be a serialized list of objects, each with a unique document_id and content field. [GENERATED_ANSWER] is the final text output from your LLM. For high-stakes domains like healthcare or finance, the evidence field is critical; it creates a direct audit trail that a human reviewer can use to quickly verify the model's judgment. If your trace data includes retrieval scores or document metadata, consider appending it to the document content to give the auditor more context, but avoid bloating the prompt with non-essential fields that could distract from the grounding task.

The Source Document Grounding Scorecard Prompt is designed to run as a post-generation evaluation step within a RAG trace pipeline. After a model produces a final answer and the trace captures the full set of retrieved documents, this prompt scores each document on two axes: usage (was the document used, partially used, or ignored in the final output?) and faithfulness (if used, did the output represent the document's content accurately?). The resulting per-document scorecard is a structured artifact that feeds directly into retrieval quality dashboards, allowing teams to monitor precision, recall, and hallucination signals at the document level rather than relying solely on aggregate answer-level metrics.

To integrate this prompt into an application harness, you must first ensure the trace contains the required inputs: the final generated output, the list of retrieved documents with their identifiers and full text, and any metadata about retrieval rank or source. The harness should iterate over each retrieved document, calling the prompt once per document or batching multiple documents into a single call if the model's context window permits. Validation is critical: parse the JSON output and confirm that each scorecard entry contains a valid usage_status (one of used, partially_used, ignored), a faithfulness object with a status field (faithful, partially_faithful, unfaithful, not_applicable), and a non-empty evidence array citing specific spans from both the document and the output. Reject malformed responses and retry with a stricter schema reminder. For high-stakes domains such as healthcare or legal review, route any document scored as unfaithful or partially_faithful to a human review queue before the scorecard is considered final.

Model choice matters for this evaluation task. Use a model with strong instruction-following and structured output capabilities (such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro) because the prompt requires precise span extraction and consistent JSON formatting. Avoid smaller or older models that may conflate partially_used with ignored or produce hallucinated evidence spans. Logging should capture the full prompt input, the raw model response, the parsed scorecard, and any validation failures for every document evaluated. Store these logs alongside the original trace so that dashboard queries can join retrieval decisions with grounding scores. Retries: if validation fails due to malformed JSON, retry once with an explicit error message appended to the prompt. If the retry also fails, flag the document as eval_error and exclude it from aggregate metrics rather than silently accepting a broken score. Tool use: this prompt does not require external tools, but the harness may call a separate span-extraction tool to verify that the cited evidence offsets actually exist in the source documents before trusting the faithfulness score.

For continuous monitoring, schedule this prompt to run on a sample of production traces (e.g., 10% of sessions or all traces flagged by a lightweight anomaly detector). Feed the resulting scorecards into a dashboard that tracks per-document grounding rates over time, broken down by retrieval source, document type, or prompt version. Set alerts on threshold breaches: for example, if the unfaithful rate exceeds 5% for any document source in a rolling window, trigger an investigation. Avoid running this prompt on every trace without sampling unless your cost and latency budget explicitly supports it—each trace may involve dozens of documents, and the evaluation cost can quickly exceed the generation cost. Start with sampled evaluation, validate that the scorecard correlates with user-reported issues, and then expand coverage based on observed failure patterns.