Citation Accuracy Gate Evaluation Prompt

This prompt is a release gate for RAG system owners and platform engineers who need an automated, structured audit of citation accuracy before promoting a new retrieval pipeline or generation prompt to production. The job-to-be-done is verifying that a generated answer is faithfully grounded in its cited sources, not performing general fact-checking or evaluating the answer's helpfulness in isolation. It assumes you already have the final generated answer and the exact source chunks provided to the model during retrieval. The output is a machine-readable audit report that measures precision, recall, hallucinated sources, and unsupported claims, enabling a data-driven promotion decision.

Use this prompt when you are modifying retrieval logic, changing embedding models, updating chunking strategies, or iterating on the generation prompt itself. It is designed to catch regressions where the model cites a source that does not support the claim, fabricates a source identifier, or makes a statement with no grounding in the provided context. The prompt is not a substitute for human review of sensitive content, nor is it designed for real-time guardrailing on every user request—its cost and latency profile make it best suited for offline evaluation runs against a golden dataset of question-answer-context triplets. Do not use this prompt to evaluate the quality of the retrieval step itself; it only judges whether the generated answer is supported by the sources it was given.

Before running this prompt, ensure you have a structured dataset where each record contains the user query, the final generated answer, and the list of source chunks with their unique identifiers. The prompt expects these as structured inputs and will return a JSON report with per-claim verification status, source-level precision/recall, and a list of unsupported or hallucinated citations. Wire this into your CI/CD pipeline as a blocking gate: if the hallucination rate exceeds your threshold or if any hallucinated source is detected, the release should be halted for human investigation. The next section provides the exact prompt template you can adapt and integrate into your evaluation harness.

This prompt template implements a citation accuracy gate for RAG systems. The model receives the original user query, the final generated answer with inline citation markers, and the full set of source chunks provided during generation. It produces a structured audit report that flags unsupported claims, hallucinated sources, and insufficient evidence. Use this as a release gate before promoting any RAG prompt to production.

codeCopy
You are a citation accuracy auditor. Your job is to verify that every citation in a generated answer points to correct and sufficient source material.

## INPUT

**User Query:**
[USER_QUERY]

**Generated Answer with Citations:**
[GENERATED_ANSWER_WITH_CITATIONS]

**Source Chunks Provided During Generation:**
[SOURCE_CHUNKS]

## TASK

For each citation marker in the generated answer, perform the following checks:

1. **Source Existence:** Does the cited source chunk actually exist in the provided source material? If a citation references a source ID or chunk number that is not present, flag it as HALLUCINATED_SOURCE.

2. **Claim Verification:** Extract the specific claim(s) associated with each citation. Compare each claim against the cited source chunk. Determine if the source:
   - FULLY_SUPPORTS the claim
   - PARTIALLY_SUPPORTS the claim
   - CONTRADICTS the claim
   - DOES_NOT_ADDRESS the claim

3. **Evidence Sufficiency:** For each claim, assess whether the cited source provides enough detail to substantiate the claim. Rate as SUFFICIENT, PARTIAL, or INSUFFICIENT.

4. **Unsupported Claims:** Identify any factual claims in the answer that have no citation marker at all. Flag these as UNCITED_CLAIM.

## OUTPUT SCHEMA

Return a JSON object with the following structure:

```json
{
  "audit_summary": {
    "total_citations": <integer>,
    "total_claims": <integer>,
    "hallucinated_sources": <integer>,
    "unsupported_claims": <integer>,
    "uncited_claims": <integer>,
    "overall_precision": <float 0-1>,
    "overall_recall": <float 0-1>,
    "passes_gate": <boolean>
  },
  "per_citation_results": [
    {
      "citation_id": "<string>",
      "citation_text": "<string>",
      "source_chunk_id": "<string or null>",
      "source_exists": <boolean>,
      "claims": [
        {
          "claim_text": "<string>",
          "verification": "FULLY_SUPPORTS | PARTIALLY_SUPPORTS | CONTRADICTS | DOES_NOT_ADDRESS",
          "evidence_sufficiency": "SUFFICIENT | PARTIAL | INSUFFICIENT",
          "source_span_evidence": "<exact text from source that supports or contradicts>"
        }
      ]
    }
  ],
  "uncited_claims": [
    {
      "claim_text": "<string>",
      "location_in_answer": "<string describing where this claim appears>"
    }
  ],
  "hallucinated_sources": [
    {
      "citation_id": "<string>",
      "referenced_source": "<the non-existent source ID or chunk reference>"
    }
  ]
}

GATE CRITERIA

Set passes_gate to true ONLY if ALL of the following are met:

• hallucinated_sources equals 0
• overall_precision >= [PRECISION_THRESHOLD]
• overall_recall >= [RECALL_THRESHOLD]
• No claim is rated CONTRADICTS
• No uncited factual claims exist

CONSTRAINTS

• Do not infer claims the answer does not explicitly make.
• If a source chunk is ambiguous, rate it as PARTIALLY_SUPPORTS and note the ambiguity.
• For source_span_evidence, quote the exact text from the source. Do not paraphrase.
• If the generated answer contains no citations at all, set total_citations to 0 and flag all factual claims as UNCITED_CLAIM.
• Do not penalize the answer for missing information the user did not request.

Adapt this template by adjusting the GATE CRITERIA thresholds to match your risk tolerance. For high-stakes domains like healthcare or legal, set PRECISION_THRESHOLD and RECALL_THRESHOLD to 1.0 and add a mandatory human review step for any CONTRADICTS finding. For lower-stakes applications, you might accept precision of 0.9 and allow PARTIALLY_SUPPORTS claims to pass. Wire the output into your CI/CD pipeline by parsing passes_gate as a boolean gate signal and surfacing per_citation_results in your evaluation dashboard for manual inspection when the gate fails.

This prompt is designed to run as an offline evaluation gate, not in the real-time RAG serving path. It should be executed after a batch of RAG responses has been generated, typically during a staging or canary evaluation phase. The prompt expects a structured input containing the original user query, the generated response with its citations, and the full source documents or passages that were retrieved. Running this in the serving path would add unacceptable latency and cost; instead, sample a representative subset of production or staging traffic and run the evaluation asynchronously.

To integrate this into a CI/CD pipeline, build a harness that: (1) loads a golden dataset of query-response-source triples or samples from a staging traffic log; (2) calls the evaluation prompt for each sample, passing the query, response, and retrieved sources as structured inputs; (3) parses the JSON output containing per-citation precision/recall scores, unsupported claim flags, and hallucinated source detection; (4) aggregates results into a pass/fail decision based on predefined thresholds (e.g., citation precision below 0.85 triggers a block). Use a model with strong instruction-following and JSON output capabilities, such as gpt-4o or claude-3-5-sonnet, and set temperature=0 for deterministic evaluation. Implement retries with exponential backoff for malformed JSON outputs, and log every evaluation result with the prompt version, model version, and dataset version for auditability.

The harness must include source span verification: for each citation in the response, extract the cited text span from the source document and compare it against what the response claims. If the prompt flags a hallucinated source (a citation that does not exist in the provided sources), the harness should automatically fail that sample and surface it for human review. For evidence sufficiency scoring, set a minimum threshold (e.g., 0.7) below which the response is considered inadequately grounded. Wire the aggregated report into your release gate scorecard, and ensure that any sample with a hallucinated source or unsupported claim count above zero triggers a blocking condition. Do not use this prompt as a substitute for human review on high-stakes domains like healthcare or legal; instead, route flagged samples to a review queue and require human sign-off before promotion.