Rationale Generation

The fundamental tension in rationale generation. Faithful rationales accurately reflect the model's true internal computation, while plausible rationales sound convincing to a human but may misrepresent the actual decision process.

• Faithfulness: Requires mechanistic interpretability or causal tracing to verify
• Plausibility: Often achieved through post-hoc rationalization by a secondary model
• A plausible but unfaithful rationale creates a false sense of security and undermines auditability

The mechanism of anchoring generated explanations to verifiable data. Evidence attribution points to specific segments of the input, while source grounding links claims to external documents or training data.

• Enables citation generation for factual assertions
• Critical for factual consistency — ensuring rationales do not contradict source material
• Transforms a rationale from a mere assertion into an auditable, verifiable artifact

Effective rationales often explain why outcome A was predicted instead of B. Contrastive explanations highlight the differentiating factors, while counterfactual rationales describe minimal input changes that would flip the prediction.

• Provides actionable explanations — users learn what to change for a different outcome
• Aligns with human psychology: people naturally ask 'why this and not that?'
• Essential for recourse mechanisms in regulated domains like lending

Eliciting and exposing intermediate reasoning steps from large language models. Chain-of-thought prompting uses few-shot examples to encourage step-by-step logical decomposition before the final answer.

• Enables inspection of the reasoning trajectory, not just the conclusion
• Supports simulatability — a human can follow the steps to anticipate the output
• Vulnerable to hallucinated reasoning that appears logical but contains fabricated steps

Rationales tailored to the recipient's expertise and delivered through dynamic interfaces. User-adaptive explanations adjust complexity for data scientists versus end-users, while interactive explanations allow follow-up probing.

• Supports minimal sufficient explanations — the shortest justification that remains complete
• Enables drilling down from summary to granular feature-level detail
• Aligns with GDPR Right to Explanation requirements for meaningful information

The capacity to express confidence levels in natural language and flag fabricated content. Verbalized uncertainty communicates model doubt, while hallucination detection identifies nonsensical or unfaithful statements.

• Faithfulness metrics quantitatively score how accurately a rationale mirrors internal logic
• Prevents overconfident delivery of incorrect justifications
• Essential for high-stakes domains where an incorrect rationale can cause real harm

The direct output of rationale generation systems—human-readable justifications produced alongside predictions. NLEs translate model logic into accessible language.

• Bridges the gap between latent model representations and human understanding
• Often evaluated on fluency, conciseness, and actionability
• Example: 'Loan denied because debt-to-income ratio of 47% exceeds the 43% threshold'

A critical distinction in rationale quality. Faithful rationales accurately reflect the model's true internal reasoning process. Plausible rationales sound convincing to humans but may misrepresent actual decision mechanics.

• Faithfulness requires causal tracing through model weights
• Plausibility alone creates audit risk in regulated industries
• Research shows standard post-hoc explainers often generate plausible but unfaithful explanations

A technique that elicits step-by-step reasoning from large language models by providing few-shot examples of intermediate logical steps. CoT prompting is the primary mechanism for rationale generation in modern LLMs.

• Improves performance on arithmetic, commonsense, and symbolic reasoning tasks
• Enables zero-shot rationale generation when combined with trigger phrases like 'Let's think step by step'
• Foundation for self-consistency decoding methods

The mechanism of anchoring generated explanations to specific segments of source input data or external documents. Critical for verifiable rationales in enterprise settings.

• Citation generation creates precise references to supporting documents
• Factual consistency metrics ensure rationales don't contradict source material
• Essential for compliance with GDPR Right to Explanation requirements

Techniques to identify generated rationales containing fabricated, nonsensical, or unfaithful information. A critical safety layer for production rationale systems.

• Faithfulness metrics automatically score alignment with internal decision processes
• Self-consistency checks compare multiple sampled rationales for contradictions
• Source verification cross-references claims against retrieved evidence

Natural language descriptions of minimal changes to an input that would have resulted in a different prediction. These explanations answer 'what would need to change?' questions.

• Example: 'If your annual income were $5,000 higher, the loan would be approved'
• Provides actionable recourse for users affected by automated decisions
• Closely related to contrastive explanations and algorithmic fairness