Evidence Citation Accuracy in CoT Prompt

This prompt is built for RAG evaluation teams who need to verify that every reasoning step in a chain-of-thought trace is properly supported by the cited evidence. It acts as an automated judge that scores citation accuracy, flags misattributions, and detects quote-context mismatches. Use this when you are shipping a reasoning system where the path matters as much as the final answer, and you cannot afford hallucinations disguised as evidence-backed logic. The ideal user is an AI engineer or evaluation lead who already has a reasoning trace with explicit citations and the corresponding source documents available for cross-referencing.

This prompt belongs in your evaluation pipeline, not in your user-facing generation flow. It assumes you have already extracted a reasoning trace from your system—complete with step-by-step logic and explicit references to source passages—and that you have the full text of those source documents available. The prompt will produce a structured scorecard that includes per-step citation accuracy labels, misattribution flags, and quote-context mismatch detection. You should wire this into a batch evaluation harness that runs after each reasoning trace is generated, logging results for trend analysis and regression detection.

Do not use this prompt when you lack access to the original source documents, when the reasoning trace does not include explicit citations, or when you need a single holistic score without step-level granularity. It is also inappropriate for evaluating creative writing, subjective analysis, or tasks where evidence support is not the primary quality dimension. If your system generates answers without showing its work, start with a groundedness or faithfulness evaluation prompt instead. For high-stakes domains like healthcare or legal review, always pair this automated judge with human spot-checking and maintain an audit trail of judge decisions.

This prompt template is the core evaluation instruction for grading evidence citation accuracy in chain-of-thought outputs. It is designed for RAG evaluation teams who need to verify that every citation in a reasoning trace points to source material that genuinely supports the claim being made. The prompt produces structured scores, misattribution flags, and quote-context mismatch detection, making it suitable for automated evaluation pipelines where citation fidelity is a release gate. Copy this template into your evaluation harness and replace the square-bracket placeholders before each run.

textCopy
You are an evidence citation auditor evaluating a reasoning trace. Your job is to verify that every cited piece of evidence actually supports the reasoning step that references it. Do not evaluate whether the final answer is correct. Only evaluate citation accuracy.

## INPUT

**Reasoning Trace:**
[REASONING_TRACE]

**Source Documents (with citation IDs):**
[SOURCE_DOCUMENTS]

## EVALUATION INSTRUCTIONS

For each citation reference in the reasoning trace, perform the following checks:

1. **Citation Existence:** Does the cited source ID exist in the provided source documents? If not, flag as `citation_not_found`.
2. **Quote Accuracy:** If the reasoning step quotes or paraphrases the source, does the quoted material actually appear in the cited source? Flag any fabricated or distorted quotes as `quote_mismatch`.
3. **Context Fidelity:** Does the cited passage, when read in its full context, support the claim made in the reasoning step? Flag cases where the citation is taken out of context or contradicts the claim as `context_mismatch`.
4. **Claim Support Strength:** Rate how strongly the cited evidence supports the reasoning step's claim on this scale:
   - `fully_supported`: The evidence directly and unambiguously supports the claim.
   - `partially_supported`: The evidence provides some support but requires inference or additional assumptions.
   - `unsupported`: The evidence does not support the claim or contradicts it.

## OUTPUT SCHEMA

Return a JSON object with this exact structure:

{
  "overall_citation_accuracy_score": <float between 0.0 and 1.0>,
  "total_citations_evaluated": <integer>,
  "citation_evaluations": [
    {
      "citation_id": "<string>",
      "reasoning_step_index": <integer>,
      "reasoning_step_text": "<excerpt from reasoning trace>",
      "citation_exists": <boolean>,
      "quote_accurate": <boolean>,
      "context_fidelity": "<accurate | mismatch | partial_mismatch>",
      "claim_support_strength": "<fully_supported | partially_supported | unsupported>",
      "explanation": "<brief explanation of the evaluation>",
      "flagged": <boolean>
    }
  ],
  "misattribution_flags": [
    {
      "citation_id": "<string>",
      "flag_type": "<citation_not_found | quote_mismatch | context_mismatch | unsupported>",
      "severity": "<critical | major | minor>",
      "detail": "<description of the problem>"
    }
  ],
  "summary": "<one-paragraph summary of citation accuracy findings>"
}

## SCORING RULES

- `overall_citation_accuracy_score` = (number of citations with `claim_support_strength` of `fully_supported` or `partially_supported` AND no flags) / `total_citations_evaluated`
- A citation with any flag (`citation_not_found`, `quote_mismatch`, `context_mismatch`) automatically counts as inaccurate for the overall score.
- `partially_supported` citations without flags count as 0.5 toward the score numerator.
- `fully_supported` citations without flags count as 1.0 toward the score numerator.

## CONSTRAINTS

- Only evaluate citations that appear in the reasoning trace. Do not invent citations to check.
- If the reasoning trace makes claims without citations, note this in the summary but do not penalize the citation accuracy score.
- Do not evaluate the quality of the reasoning itself, only the accuracy of evidence citations.
- If a source document is ambiguous, note the ambiguity in your explanation and score conservatively.
- [ADDITIONAL_CONSTRAINTS]

## EXAMPLES

[FEW_SHOT_EXAMPLES]

After copying the template, replace the placeholders with your actual data. [REASONING_TRACE] should contain the full chain-of-thought output you are evaluating, with citation markers preserved. [SOURCE_DOCUMENTS] must include the complete text of every retrievable source, each labeled with the same citation IDs used in the reasoning trace. Use [ADDITIONAL_CONSTRAINTS] to inject domain-specific rules, such as requiring exact quote matching for regulated industries or adjusting the partial-support threshold. [FEW_SHOT_EXAMPLES] is critical for calibration: include at least two examples showing correct citation evaluation and one showing a misattribution flag. Without examples, the model may conflate reasoning quality with citation accuracy or miss subtle context mismatches. Before running this in production, validate the output JSON against the schema and confirm that every citation in the reasoning trace appears in the evaluation output. Missing citations in the evaluation indicate the model skipped a check, which is a silent failure mode.

The Evidence Citation Accuracy prompt is designed to operate as a post-hoc evaluation step in a RAG pipeline, not as a real-time guardrail. After a reasoning trace is generated, this prompt should be called asynchronously to score each citation against its source evidence. The harness must pass the full reasoning trace, the list of cited evidence chunks with their unique identifiers, and the original source documents. The model returns a structured JSON object containing per-citation accuracy scores, misattribution flags, and quote-context mismatch descriptions. This output should be logged alongside the original trace for auditability and used to gate whether the reasoning output is surfaced to users or routed for human review.

Wire the prompt into an evaluation service that runs after the primary reasoning model returns. The service should: (1) extract all citation references from the reasoning trace using a regex or parser that matches your citation format (e.g., [1], [doc_3]); (2) retrieve the corresponding evidence chunks from your vector store or document store using the citation IDs; (3) assemble the prompt with the reasoning trace, the evidence chunks, and the source documents; (4) call a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent); (5) parse the JSON response and validate that every cited reference in the trace received a score. If the model returns malformed JSON, implement a single retry with the error message appended to the prompt. If the retry also fails, flag the entire trace for human review and log the raw response. For high-stakes domains like healthcare or legal, set a threshold—any citation with an accuracy score below 0.8 or any misattribution flag set to true should automatically escalate the output to a human reviewer before it reaches the end user.

Model choice matters here. Use a model with a large context window (128k tokens or more) to accommodate long reasoning traces with many citations. Avoid models that struggle with fine-grained text comparison, as the core task requires precise alignment between a reasoning step and a specific passage. If your RAG system uses chunked evidence, ensure each chunk passed to the evaluator includes enough surrounding context to resolve anaphora and implicit references. Log every evaluation result with the trace ID, timestamp, model version, and scores. This log becomes your audit trail for citation accuracy over time and feeds into regression tests when you change your retrieval or reasoning prompts. Do not use this prompt to block outputs in latency-sensitive user-facing flows; it is an evaluation and gating tool, not a streaming guardrail.