Post-Hoc Rationalization

The generated explanation does not reflect the model's true computational process. Instead, it is a confabulation—a coherent, internally consistent, but factually incorrect narrative. The model acts as an involuntary storyteller, inventing a logical-sounding reason that aligns with the final output but is disconnected from the actual feature weights or attention patterns that drove the prediction. This fundamentally undermines trust in automated decision-making.

The defining structural flaw: the decision precedes the rationale. The model's forward pass computes an output based on complex, often opaque, statistical correlations. Only after this output is locked in does a secondary process—often the same model in a generative mode—construct a verbal justification. This reverses the human expectation that reasoning leads to a conclusion, making the explanation a backward-facing narrative rather than a causal trace.

The generated rationale is dangerously effective because it is optimized for human plausibility, not mechanistic accuracy. Large language models are adept at producing text that conforms to human expectations of logical structure. This fluency creates an illusion of explanatory depth, where a well-articulated but false reason is more convincing to a human auditor than a true but messy or complex one, effectively masking the model's brittle reliance on spurious correlations.

Post-hoc rationalization is the direct opposite of a Faithful Chain-of-Thought. A faithful CoT trace is a causal map of the computation; the reasoning steps are the mechanism by which the answer is derived. In post-hoc rationalization, the reasoning steps are a post-hoc narrative decoration. Key differentiators include:

• Causal Test: Intervening on the rationale would not change the original decision.
• Origin: It is generated after the final answer logits are computed.

This phenomenon is a direct manifestation of the Clever Hans effect in language models. Just as the horse learned to read unconscious human cues rather than perform arithmetic, a model can latch onto spurious statistical shortcuts in its training data to make correct predictions. When asked to explain a correct prediction, it will generate a plausible, domain-relevant justification, completely masking the fact that its true heuristic was a superficial pattern match on an irrelevant feature like text formatting or a keyword bias.

The primary method for unmasking post-hoc rationalization is causal intervention. This involves modifying the model's internal state or input to test the explanation's validity:

• Activation Patching: Replacing the activations associated with the 'reason' to see if the output changes.
• Counterfactual Input Editing: Changing the feature cited in the explanation. If the model's output remains unchanged, the explanation was a fabrication.
• Process Supervision: Training a reward model to score the logical validity of each step, not just the final outcome, to penalize confabulated chains.

A reasoning trace that accurately reflects the true causal process by which the model arrived at its final answer. Unlike post-hoc rationalization, a faithful chain-of-thought is free from confabulation and represents the model's actual computation.

• Key distinction: Faithfulness is about causal accuracy, not just logical coherence
• Testing method: Use activation patching to verify if the reasoning steps causally influence the output
• Failure mode: A logically valid rationale can still be unfaithful if the model used different heuristics

A model's reliance on spurious statistical correlations in training data to make correct predictions for the wrong reasons. Named after the horse that appeared to perform arithmetic but was actually reading unconscious cues from its handler.

• Connection to post-hoc rationalization: The model generates a plausible explanation that references valid features while its true decision boundary relies on confounding variables
• Detection: Use counterfactual testing and background removal to isolate true drivers
• Enterprise risk: Models passing validation tests while using non-robust features that fail in production

A training methodology that provides feedback on each intermediate step of a model's reasoning chain, rewarding correct logical progression rather than just the final outcome. This directly combats post-hoc rationalization by forcing the model to learn genuine reasoning.

• Contrast with outcome supervision: Only rewards final answer correctness, which incentivizes rationalization
• Implementation: Requires human annotators or a process reward model to label step-level correctness
• Result: Models trained with process supervision produce more faithful and interpretable reasoning traces

A cascading failure mode where an initial factual error in a reasoning chain leads to a sequence of subsequent errors. The model builds further logic on an incorrect premise, compounding the original mistake.

• Relationship to post-hoc rationalization: An initial confabulated justification can trigger a snowball effect
• Mechanism: Each step conditions on the previous, so errors propagate autoregressively
• Mitigation: Chain-of-verification and external tool grounding can arrest the cascade early

A quantitative score designed to measure the degree to which a generated reasoning trace accurately represents the model's true computational process. These metrics are essential for auditing whether an explanation is genuine or a post-hoc fabrication.

• Approaches: Causal intervention tests, counterfactual consistency checks, and feature attribution alignment
• Key challenge: Distinguishing between a faithful rationale and a highly persuasive but causally disconnected one
• Use case: Continuous monitoring of LLM reasoning quality in production systems

The theory that neural networks represent more independent features than they have dimensions in a given layer by encoding them in overlapping, nearly orthogonal directions. This creates the conditions for post-hoc rationalization.

• Mechanism: A single neuron can participate in multiple feature representations, making it impossible to attribute a decision to a single clean concept
• Implication: Linear probing may recover plausible features that weren't causally active
• Research frontier: Sparse autoencoders attempt to disentangle these superimposed features into monosemantic components