LLM-as-Explainer

The LLM acts as a secondary system, analyzing the input-output pairs of a primary black-box model to generate a plausible justification after the prediction is made. This decouples the explainer from the original model's architecture, allowing it to explain legacy or opaque systems without modifying their weights. The LLM leverages its world knowledge and linguistic priors to construct a coherent narrative, though this introduces a risk of generating plausible but unfaithful rationales that sound correct but misrepresent the actual decision logic.

Instead of treating the LLM as an external observer, this paradigm uses few-shot prompting to force the model to verbalize its own step-by-step reasoning before delivering a final answer. By structuring prompts with intermediate reasoning examples, the LLM externalizes its internal computation into a human-readable trace. This technique is foundational for self-explaining systems where the same model handles both task completion and rationale generation, improving simulatability and enabling human auditors to pinpoint logical errors in the reasoning chain.

A critical capability where the LLM explicitly links claims in its generated rationale to specific segments of the source input or external documents. This moves beyond free-text justification to citation-backed explanations that can be verified. Techniques include:

• Direct quote extraction from the input context
• Document retrieval with pointer generation
• Factual consistency scoring against a knowledge base Grounding is essential for high-stakes domains like legal analysis and medical diagnosis where hallucinated justifications are unacceptable.

The LLM generates rationales structured around why A instead of B, explaining the minimal necessary conditions that led to a specific outcome. This includes producing counterfactual rationales—natural language descriptions of the smallest input changes that would flip the prediction. For example, 'The loan was denied because your debt-to-income ratio was 43%. If it were below 36%, the application would have been approved.' This framing directly supports actionable explanations and algorithmic recourse, giving end-users a clear path to change future outcomes.

A central tension in LLM-as-Explainer systems. Plausible rationales are human-acceptable and linguistically coherent but may not reflect the model's true internal computation. Faithful rationales accurately mirror the actual decision logic, even if they are less fluent. LLMs are naturally biased toward plausibility due to their training on human-authored text. Mitigation strategies include:

• Faithfulness metrics that compare generated rationales against perturbation-based importance scores
• Self-consistency checks across multiple reasoning paths
• Adversarial rationale testing to detect fabricated justifications

The LLM dynamically tailors explanation depth, vocabulary, and structure based on the end-user's role and expertise. A data scientist receives a technical rationale referencing SHAP values and attention weights, while a consumer receives a plain-language summary. This paradigm supports multi-turn explanatory dialogues where users can ask follow-up questions like 'Why was this feature more important than that one?' or 'What does this term mean?' The LLM maintains conversational context, enabling progressive disclosure of complexity and truly interactive interpretability.

Human-readable justifications generated alongside a prediction. Unlike feature attribution scores, NLEs articulate the why in plain language. An LLM-as-Explainer produces NLEs by conditioning on input features and the target model's output. Key properties include readability, completeness, and actionability. Example: 'Loan denied because your debt-to-income ratio of 45% exceeds the 36% threshold.'

A critical distinction in explainability. Faithful rationales accurately reflect the model's true internal reasoning. Plausible rationales sound convincing to humans but may misrepresent actual decision mechanics. LLM-as-Explainer often generates plausible rationales by default. Achieving faithfulness requires:

• Grounding in attention weights or gradients
• Constraining generation to observed feature values
• Verifying via simulatability tests

A technique that elicits step-by-step reasoning from LLMs by providing few-shot examples of intermediate logical steps. When used for explainability, CoT prompting forces the model to externalize its reasoning trace. This trace serves as a rationale. Key variants:

• Zero-shot CoT: Simply appending 'Let's think step by step'
• Auto-CoT: Automatically generating diverse reasoning chains
• Self-consistency: Sampling multiple chains and taking the majority conclusion

The process of linking claims within a generated rationale to verifiable external documents or specific training data points. Without grounding, LLM-generated explanations risk hallucination. Grounding mechanisms include:

• Retrieval-Augmented Generation (RAG): Fetching relevant documents before generation
• Citation generation: Producing precise references to source passages
• Evidence attribution: Pointing to specific input segments that support each claim

Natural language descriptions of minimal input changes that would alter a prediction. Example: 'Your loan would be approved if your credit score increased by 15 points.' These rationales provide recourse—actionable steps for users. LLMs excel at generating counterfactuals because they can reason about hypothetical scenarios and articulate them fluently. Key challenge: ensuring counterfactuals are minimal and feasible.

Techniques to identify generated explanations containing fabricated or unfaithful information. LLM-as-Explainer is susceptible to hallucination because the model may invent plausible-sounding but false justifications. Detection methods:

• Factual consistency checks against source data
• SelfCheckGPT: Sampling multiple responses to detect inconsistencies
• Entailment models: Verifying if the rationale logically follows from the evidence