Explanation Faithfulness

The explanation must reflect the true causal mechanisms the model used, not just correlated features. A faithful rationale identifies the input tokens or features that actively caused the output through the model's internal weights.

• Key Test: Perturbing the identified features must change the prediction in the way the explanation describes.
• Contrast: A plausible explanation might cite 'positive sentiment words,' while a faithful one identifies the specific attention head that amplified a negation phrase.

A faithful explanation must account for the entirety of the model's decision logic, not just a convenient subset. It should capture both the primary signal and any suppressing or inhibitory factors.

• Sufficiency: The explanation alone should provide enough information to simulate the model's output.
• No Hidden Factors: If a model relies on a spurious correlation (e.g., background pixels), a faithful explanation must reveal this, even if it exposes model flaws.

The rationale must be traceable to specific internal model components—attention patterns, neuron activations, or circuit motifs—rather than being generated by a separate, disconnected explainer model.

• Implementation: Achieved through mechanistic interpretability techniques like circuit discovery or logit attribution.
• Failure Mode: Post-hoc rationalization by an external LLM often fails this test, as the explainer model hallucinates a logical story disconnected from the original model's weights.

A faithful explanation must hold under minimal counterfactual edits. If the explanation claims a specific feature was decisive, removing or negating that feature should flip the prediction.

• Contrastive Explanations: Faithful systems can accurately answer 'Why A instead of B?' by identifying the minimal necessary conditions that differentiate the outcomes.
• Stability: The explanation should not change dramatically with minor, semantically irrelevant input perturbations.

The explanation must not simply restate the output in different words. A faithful rationale provides independent evidence for the decision, grounded in the input data and model internals, not the prediction itself.

• Example of Circularity: 'The loan was denied because the applicant was high-risk.'
• Faithful Alternative: 'The loan was denied because the debt-to-income ratio exceeded 43% and the credit history length was below 24 months, which activated neuron cluster 452 in layer 8.'

A human observer, given the explanation and the input, should be able to correctly anticipate the model's output on a new, unseen example. This is the ultimate behavioral test of faithfulness.

• Metric: Forward simulation accuracy—how often a human's predicted output matches the model's actual output when guided only by the explanation.
• Practical Use: Essential for auditors who need to verify that the documented logic is the logic actually executed in production.

A generated explanation that accurately mirrors the model's true internal reasoning process. Unlike plausible rationales, faithful rationales do not fabricate a convincing but false story. They are verified through causal interventions—if you remove the cited features, the model's prediction must change accordingly. Key properties:

• Must pass simulatability tests
• Grounded in actual feature weights or attention patterns
• Distinguishable from post-hoc confabulation

A human-acceptable explanation that sounds convincing but may not reflect the model's actual decision logic. These rationales prioritize human interpretability over mechanistic accuracy. A model might cite a dog's floppy ears as the reason for classifying a breed, when it actually relied on background pixels. Critical distinction: Plausibility ≠ Faithfulness. A perfectly plausible rationale can be entirely unfaithful.

The technique of generating an explanation after the model has already made its prediction, often using a secondary explainer model. This introduces a fundamental fidelity risk: the explainer may invent a coherent narrative that the primary model never used. Common approaches:

• Training a separate language decoder on model latents
• Using LLM-as-Explainer paradigms
• Surrogate model approximations

A rigorous metric evaluating whether a human observer can use the model's explanation to correctly anticipate its output on new, unseen inputs. If an explanation is truly faithful, it should enable forward simulation. Testing protocol:

1. Present explanation to human evaluator
2. Show new input without model output
3. Evaluator predicts model behavior
4. Measure prediction accuracy against ground truth

Quantitative measures designed to automatically score how accurately a generated explanation reflects the model's internal decision process. These metrics move beyond human judgment to provide objective fidelity assessments. Key approaches:

• Sufficiency: Does the explanation alone produce the same prediction?
• Comprehensiveness: Does removing explained features change the output?
• Correlation with gradient-based importance scores

Techniques used to identify and flag generated explanations that contain fabricated, nonsensical, or unfaithful information. In the context of rationale generation, hallucination manifests as citing evidence that doesn't exist in the input or asserting causal relationships the model never computed. Detection methods:

• Cross-referencing against attention weights
• Input perturbation consistency checks
• Entropy-based uncertainty signals