Post-Hoc Rationalization

A secondary surrogate model is trained specifically to mimic the primary black-box model's decision boundary in a local region. This explainer model is inherently interpretable (e.g., a linear model or decision tree) and is used to generate the rationale. The process involves:

• Perturbation: Generating variations of the input instance.
• Labeling: Querying the black-box model for predictions on these variations.
• Training: Fitting the interpretable surrogate on the perturbed dataset, weighted by proximity to the original instance. The resulting explanation is an approximation of the local logic, not a direct readout of the primary model's internal weights.

A central tension in post-hoc rationalization is between faithfulness (how accurately the explanation reflects the model's true computational process) and plausibility (how convincing the explanation is to a human).

• Faithful Rationales: May be complex, non-intuitive, or expose model reliance on spurious correlations, making them less satisfying to users.
• Plausible Rationales: Often tell a clean, logical story that aligns with human expectations but may fabricate a justification that has little to do with the model's actual feature weights. Post-hoc methods inherently risk generating plausible but unfaithful explanations because they are separate from the model's core inference logic.

Post-hoc rationalization typically operates on a local level, explaining a single prediction rather than the entire model's global behavior. The explanation is valid only in the immediate neighborhood of the input instance. Key implications include:

• Non-Generalizable: The rationale for one prediction cannot be assumed to apply to another, even a similar one.
• Boundary Sensitivity: The explanation can change drastically near decision boundaries, revealing instability in the model's logic. This locality is both a strength (precision) and a weakness (lack of a holistic model understanding).

The most common output of a post-hoc rationalization is a feature importance vector, which quantifies the contribution of each input token, pixel, or tabular feature to the final prediction. This is often visualized as:

• Saliency Maps: Highlighting influential regions in an image or text.
• Attribution Scores: Numerical weights (e.g., SHAP values) showing positive or negative influence. The rationalization engine then translates these mathematical attributions into a natural language justification, a process that introduces a second layer of potential unfaithfulness.

A specific form of post-hoc rationalization that answers: 'What minimal change would alter the outcome?' The system searches for the closest counterfactual instance that flips the prediction. This is inherently actionable. The rationale is structured as:

• Contrastive Statement: 'The loan was denied because your debt-to-income ratio was 42%. If it were below 40%, the loan would have been approved.'
• Feasibility Constraints: The search for a counterfactual must respect immutable features (e.g., age cannot decrease) and actionable ones. This method directly supports the GDPR Right to Explanation by providing a path to recourse.

Generating a post-hoc rationale adds significant computational cost to the inference pipeline. Unlike self-explaining models, this is a two-step process: primary prediction followed by explanation generation. The overhead depends on the method:

• Model-Agnostic Methods (LIME, SHAP): Require thousands of perturbed samples and model queries per explanation, leading to high latency.
• Gradient-Based Methods: Faster, as they use backpropagation, but are restricted to differentiable models.
• LLM-as-Explainer: Incur the cost of a second large model call, which can be substantial in terms of both time and compute resources.

A critical distinction in explanation quality. Faithful rationales accurately reflect the model's true internal computational logic, even if that logic is messy or non-intuitive. Plausible rationales sound convincing to humans but may be confabulations that misrepresent how the decision was actually made. Post-hoc methods risk generating plausible-but-unfaithful explanations, especially when using a secondary explainer model that has no access to the original model's weights.

A technique for eliciting step-by-step reasoning from large language models by providing few-shot examples of intermediate logical steps. Unlike post-hoc rationalization—which explains after the fact—chain-of-thought interleaves reasoning with prediction, making the rationale part of the generation process. This often yields more faithful explanations but requires the model to expose its reasoning during inference.

The mechanism of grounding generated explanations by explicitly pointing to specific segments of source input data as proof. Key techniques include:

• Citation generation: Creating precise references to source documents
• Factual consistency: Ensuring rationales don't contradict source data
• Hallucination detection: Flagging fabricated information in explanations Without grounding, post-hoc rationales risk becoming detached from the evidence that actually drove the prediction.

The regulatory requirement under the General Data Protection Regulation for providing meaningful information about the logic involved in automated decisions. Post-hoc rationalization is often deployed as a compliance mechanism, but legal scholars debate whether generated explanations satisfy the regulation's requirement for transparency into actual decision logic. Faithfulness becomes a legal concern, not just a technical one.

Rationales that not only explain a decision but also provide the user with clear steps to change the outcome in the future. For example, a loan denial explanation might specify: 'Your application was denied because your debt-to-income ratio exceeded 43%. Reducing your revolving credit balance by $5,000 would likely result in approval.' This goes beyond post-hoc justification into recourse generation.

The paradigm of using a large language model's generative capabilities to produce post-hoc rationales for itself or other black-box systems. This approach leverages the fluency of LLMs but introduces faithfulness risks: the explainer LLM may generate coherent-sounding justifications that have no causal connection to the original model's decision process. Research focuses on constraining generation with attention maps and activation data.