Source Attribution Completeness Check Prompt

This prompt is designed for RAG developers and quality engineers who need to validate citation coverage in production traces. It reviews a single trace containing the user query, the generated answer, and all retrieved source chunks. The prompt extracts every factual claim from the answer, maps each claim to supporting citations, and flags claims that lack source backing. The output is an attribution coverage score with specific locations of unsupported statements. Use this prompt when you need to quantify how much of a generated answer is grounded in retrieved evidence, when you are debugging hallucination complaints, or when you are building automated quality gates that reject answers with insufficient citation coverage.

This prompt assumes the trace already contains the full retrieved context and the final answer with inline citations. It does not perform retrieval itself and does not evaluate whether citations are correctly placed, only whether they exist for each claim. The ideal user is someone who already has access to production trace data—either through an observability platform, logging pipeline, or manual trace export—and needs a structured, repeatable way to assess citation completeness at scale. The prompt works best when the answer contains explicit citation markers (e.g., [1], [2]) that reference specific source chunks, but it can also operate on answers without inline citations by performing claim-to-source semantic matching.

Do not use this prompt when you need to verify that a citation actually supports the claim it is attached to—that requires a separate grounding verification prompt that checks the content of the cited source against the claim text. Do not use this prompt when you need to evaluate retrieval quality itself; it only assesses whether claims have citations, not whether the right documents were retrieved. Do not use this prompt on traces where the retrieved context is missing or incomplete, as the coverage score will be artificially low and misleading. For high-stakes domains like healthcare, legal, or finance, always pair this prompt with human review of flagged unsupported claims before taking action on the output.

This prompt template is designed to be dropped directly into your trace analysis pipeline. It instructs the model to act as an attribution auditor, comparing every factual claim in a generated answer against the source citations provided in the production trace. The output is a structured coverage report, not a free-text critique, making it suitable for automated downstream processing, dashboarding, or triggering a human review queue.

textCopy
You are an attribution auditor. Your task is to review a production trace from a RAG system and determine whether every factual claim in the generated answer is supported by a corresponding source citation.

## INPUT

**User Query:**
[USER_QUERY]

**Generated Answer:**
[GENERATED_ANSWER]

**Retrieved Sources (with citation IDs):**
[RETRIEVED_SOURCES]

## INSTRUCTIONS

1.  **Extract Claims:** Decompose the Generated Answer into a list of discrete, verifiable factual claims. Ignore opinions, stylistic phrasing, and purely transitional language.
2.  **Match Claims to Sources:** For each claim, search the Retrieved Sources for evidence that directly supports it. A claim is supported only if a source explicitly states the fact or the fact can be directly and unambiguously inferred from the source.
3.  **Flag Unsupported Claims:** Identify any claim that cannot be matched to a source. Flag it as "unsupported."
4.  **Flag Contradicted Claims:** Identify any claim that is directly contradicted by a source. Flag it as "contradicted" and cite the contradicting source.
5.  **Calculate Coverage Score:** Calculate an attribution coverage score as `(Number of Supported Claims) / (Total Number of Claims)`.

## OUTPUT_SCHEMA

You must respond with a single JSON object conforming to this schema:

{
  "total_claims": <integer>,
  "supported_claims": <integer>,
  "unsupported_claims": <integer>,
  "contradicted_claims": <integer>,
  "attribution_coverage_score": <float between 0.0 and 1.0>,
  "claim_details": [
    {
      "claim_text": "<exact text of the claim>",
      "status": "supported | unsupported | contradicted",
      "source_citation_ids": ["<list of citation IDs>"],
      "rationale": "<brief explanation of the match or mismatch>"
    }
  ]
}

## CONSTRAINTS

- Base your analysis strictly on the provided text. Do not use outside knowledge.
- If a claim is a paraphrase of a source, it is supported. If it adds new information not in the source, it is unsupported.
- If the Generated Answer contains no factual claims, return an empty `claim_details` array and a score of 1.0.

To adapt this prompt for your specific RAG pipeline, replace the [USER_QUERY], [GENERATED_ANSWER], and [RETRIEVED_SOURCES] placeholders with data from your trace. The [RETRIEVED_SOURCES] should be formatted as a clear mapping of citation IDs to their full text. For high-stakes domains like healthcare or finance, the output of this prompt should be treated as a diagnostic signal, not a final verdict. Always route traces with an attribution_coverage_score below your defined threshold (e.g., 0.9) to a human review queue for manual verification before any action is taken on the information.

This prompt is designed to run as a post-hoc analysis step in a RAG observability pipeline, not as a real-time guardrail in the user-facing path. It consumes a production trace containing the user query, the final generated answer, and the full list of retrieved source chunks with their identifiers. The output is a structured attribution coverage report that can be logged, aggregated, and used to trigger alerts or re-indexing jobs. Because the analysis is computationally intensive and requires the full context window, it is best executed asynchronously after the user response has been delivered.

To wire this into an application, extract the required inputs from your tracing system (e.g., LangSmith, Arize, or a custom trace store). The [TRACE_INPUT] should be a serialized JSON object containing three fields: query, answer, and sources (an array of objects with source_id and content). Validate the input shape before calling the model—reject traces where the answer is empty or the sources array is missing. On success, parse the model's JSON output and validate that the coverage_score is a float between 0.0 and 1.0, that unsupported_claims is an array, and that each claim includes a claim_text and location_in_answer. If validation fails, retry once with a stricter output schema instruction; if it fails again, log the raw output and flag the trace for manual review. For high-stakes domains like healthcare or legal, always route traces with a coverage_score below a configurable threshold (e.g., 0.8) to a human review queue before any downstream action is taken.

Model choice matters here. Use a model with strong instruction-following and JSON output capabilities, such as gpt-4o or claude-3-5-sonnet, because the task requires precise claim extraction and careful comparison against source text. Avoid smaller or older models that may conflate claims or hallucinate source content during the verification step. Log the coverage_score, the count of unsupported claims, and the specific source_ids that were missing as structured metrics in your observability platform. Over time, aggregate these metrics by knowledge base partition, retrieval configuration, or prompt version to identify systemic attribution gaps. Do not use this prompt as a substitute for human review in regulated workflows; it is a diagnostic tool that surfaces potential problems, not a final arbiter of factual accuracy.