Faithful CoT

The reasoning trace must accurately reflect the true causal process by which the model arrived at its answer. This means the generated tokens correspond to the actual computational steps, not a confabulated story invented after the fact.

• Counterfactual test: If a step in the trace is altered, the final output must change predictably
• Contrast with post-hoc rationalization: A plausible-sounding explanation generated after a decision is made, which masks the true underlying heuristics
• Key challenge: Models can generate coherent reasoning that sounds correct but bears no causal relationship to their internal computation

A faithful CoT trace captures all reasoning steps that materially contributed to the final answer, without omitting critical inferences or skipping logical dependencies.

• No hidden leaps: Every inferential jump must be explicitly articulated in the trace
• Granularity requirement: Steps must be decomposed to a level where each is individually verifiable
• Failure mode: Hallucination snowballing—when an omitted or incorrect early step cascades into a chain of errors built on a faulty premise

Each reasoning step must follow deductively or inductively from the preceding steps and the provided context. The chain must form a valid logical argument, not a series of loosely associated statements.

• Transitive closure: If A implies B and B implies C, the trace must explicitly connect A to C
• No circular reasoning: Conclusions cannot be used as premises for themselves
• Contrast with the Clever Hans Effect: A model may produce correct answers using spurious correlations rather than valid logic—a faithful trace exposes this by revealing broken logical links

Every factual claim in the reasoning chain must be directly attributable to the provided input context, retrieved documents, or verifiable external tools—not hallucinated from parametric knowledge.

• Citation integrity: Claims must reference specific spans of source text or tool outputs
• Tool-augmented reasoning: When external tools like calculators or search engines are used, the trace must show the exact query and result
• Distinction from parametric confabulation: The model must not invent facts that sound plausible but cannot be sourced from the provided context

Each intermediate step must be independently evaluable as correct or incorrect, enabling process supervision rather than outcome-only feedback.

• Step-level scoring: A Process Reward Model (PRM) can assign a correctness score to each reasoning step
• Contrast with outcome supervision: Feedback based solely on final answer correctness cannot distinguish a lucky guess from sound reasoning
• Implementation: Training with process supervision rewards correct logical progression, producing models whose CoT traces are more faithful by construction

A faithful reasoning trace exhibits predictable behavioral changes when specific steps are surgically modified or ablated—a property validated through causal intervention techniques.

• Activation patching: Replacing internal activations at specific reasoning steps should alter downstream outputs in ways consistent with the trace
• Logit lens analysis: Applying the unembedding matrix to intermediate residual stream activations should reveal predictions that align with the stated reasoning
• Counterfactual editing: Changing a premise in the trace should produce the logically entailed change in the conclusion, not an arbitrary or unrelated shift