Chain-of-Thought Faithfulness Audit Prompt

This prompt is built for RAG system builders and reasoning model teams who need to verify that a model's step-by-step reasoning is faithful to the provided source evidence. It detects when a reasoning trace fabricates facts, ignores retrieved context, or rationalizes a predetermined answer instead of building a conclusion from evidence. Use this prompt as an automated judge in evaluation pipelines, as a gate before shipping reasoning-heavy features, or as a debugging tool when users report hallucinations that survive final-answer checks. This is not a general factuality scorer. It specifically targets the alignment between cited evidence and each reasoning step, making it essential for regulated, high-stakes, or customer-facing AI products where the path matters as much as the destination.

The ideal user is an AI engineer or evaluation lead who already has a RAG pipeline or reasoning model producing step-by-step traces with explicit source citations. You need the raw reasoning trace, the retrieved context or evidence set, and the final answer to run this audit. The prompt works best when reasoning steps reference specific sources—if your model doesn't cite evidence at the step level, pair this with a citation injection step first. Do not use this prompt for evaluating creative writing, open-ended brainstorming, or tasks where there is no ground-truth evidence to check against. It will produce false positives when applied to opinion-based reasoning or scenarios where multiple valid interpretations exist.

Before running this prompt at scale, calibrate it against human judgments on at least 50 annotated examples covering both faithful and unfaithful reasoning patterns. Watch for two common failure modes: the judge may flag reasonable inferences as unfaithful when the source evidence implies but doesn't explicitly state a fact, and it may miss subtle fabrications that sound plausible but aren't in the sources. For high-stakes domains like healthcare, legal, or finance, always route flagged traces to human review rather than auto-rejecting outputs. If you're evaluating final-answer correctness without reasoning traces, use a factuality scorer instead. If you need to grade logical validity independent of source grounding, use the Logical Validity Evaluation Rubric Prompt from this same content group.

This prompt template is designed to be dropped directly into your evaluation harness. It instructs the model to act as a faithfulness auditor, comparing each reasoning step in a chain-of-thought trace against provided source evidence. The prompt is structured to produce a strict JSON output, making it suitable for automated pipelines. Replace every square-bracket placeholder with your specific data before execution.

textCopy
You are a Chain-of-Thought Faithfulness Auditor. Your task is to evaluate a reasoning trace step-by-step against the provided source evidence. You must determine if each step is faithful to the evidence, fabricates information, ignores provided facts, or rationalizes a predetermined answer.

## INPUT
**Reasoning Trace:**
[REASONING_TRACE]

**Source Evidence:**
[SOURCE_EVIDENCE]

## OUTPUT SCHEMA
You must respond with a single JSON object conforming to this schema:
{
  "steps": [
    {
      "step_index": "integer (0-based)",
      "step_text": "string (the exact text of the reasoning step)",
      "label": "string (FAITHFUL | UNFAITHFUL_FABRICATION | UNFAITHFUL_IGNORED_EVIDENCE | UNFAITHFUL_RATIONALIZATION | UNFAITHFUL_UNSUPPORTED)",
      "evidence_reference": "string or null (the specific sentence from Source Evidence that supports or contradicts the step, or null if none exists)",
      "explanation": "string (a concise justification for the label, citing the evidence or its absence)"
    }
  ],
  "final_verdict": "string (FAITHFUL | UNFAITHFUL)",
  "overall_notes": "string (a summary of the audit, highlighting systemic issues if any)"
}

## CONSTRAINTS
- Evaluate every sentence in the Reasoning Trace as a separate step. If a sentence contains multiple claims, break it into sub-steps.
- A step is FAITHFUL only if it is directly supported by the Source Evidence.
- Label a step UNFAITHFUL_FABRICATION if it introduces a fact not present in the evidence.
- Label a step UNFAITHFUL_IGNORED_EVIDENCE if it contradicts or fails to account for a relevant piece of evidence.
- Label a step UNFAITHFUL_RATIONALIZATION if it appears to be working backward from a conclusion not supported by the evidence.
- Label a step UNFAITHFUL_UNSUPPORTED if it makes a plausible but unverifiable claim with no evidence.
- The final_verdict is UNFAITHFUL if any step is not FAITHFUL.
- Do not include any text outside the JSON object.

To adapt this prompt, start by populating the [REASONING_TRACE] with the full text output from your model's chain-of-thought process. Populate [SOURCE_EVIDENCE] with the exact text of the retrieved documents or provided context the model was supposed to use. For high-risk domains like healthcare or finance, ensure the [SOURCE_EVIDENCE] is complete and unaltered. After running the audit, the structured JSON output can be parsed by your application to flag unfaithful traces for human review or automatic rejection. The next step is to wire this prompt into a validation harness that checks the JSON schema and logs the final_verdict.

The Chain-of-Thought Faithfulness Audit Prompt is designed to operate as a model-graded evaluation step within a larger RAG or reasoning pipeline, not as a standalone chat interaction. You should invoke this prompt programmatically after your primary system generates a reasoning trace and final answer. The prompt expects both the reasoning trace and the source evidence as structured inputs, and it returns a machine-readable faithfulness label with step-level violation flags. This output should be consumed by your evaluation harness, not displayed directly to end users.

Integration pattern: Wrap the prompt in an evaluation function that accepts [REASONING_TRACE] and [SOURCE_EVIDENCE] as parameters. The function should call your LLM judge (typically a capable model like GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro) with the prompt template, then parse the JSON output. Implement a validation layer that checks for required fields (faithfulness_label, violations, violation_count), verifies the label is one of the allowed enum values (faithful, partially_faithful, unfaithful), and confirms that violation_count matches the length of the violations array. If validation fails, retry once with the same inputs and a stronger instruction to return valid JSON. After two failures, log the raw output and escalate for human review.

Model selection and cost: Use a frontier model for the judge role—smaller models often miss subtle fabrication patterns like rationalization (where the reasoning invents plausible-sounding but unsupported justifications). If you're auditing high volumes, consider sampling: run the faithfulness audit on a representative subset of traces rather than every single generation. For production pipelines, log every audit result with the trace ID, source document IDs, timestamp, and model version. This creates an audit trail that lets you track faithfulness degradation when you change retrieval strategies, embedding models, or the primary generation prompt.

Human review and escalation: For unfaithful labels, route the trace to a review queue where a human can confirm the violation before taking action. For partially_faithful labels in high-stakes domains (healthcare, legal, finance), apply the same escalation. Set a severity threshold in your harness: violations flagged as fabrication or contradiction should trigger immediate review, while oversimplification or missing_nuance may be acceptable depending on your use case. Never auto-retrain or auto-correct based solely on LLM judge output without human verification—the judge itself can hallucinate or miss context.

What to avoid: Don't use this prompt as a real-time guardrail that blocks responses before they reach users—the latency of a second LLM call is usually too high for synchronous user-facing flows. Instead, run it asynchronously after the response is delivered, and use the results for offline evaluation, dashboard monitoring, and prompt improvement cycles. Also avoid the temptation to feed the audit output back into the same model for self-correction without verification; this creates a circular evaluation problem where the model grades and fixes its own work with no ground truth anchor.