Simulatability

Simulatability is fundamentally an extrinsic evaluation metric, requiring human subjects to perform a prediction task. The core protocol involves:

• Providing a user with the model's input data and its corresponding explanation (e.g., feature attributions, a rule, or a counterfactual).
• Asking the user to predict the model's output based solely on that information.
• Measuring the accuracy of the human's predictions against the model's actual outputs. High simulatability scores indicate the explanation successfully conveyed the model's decision logic, enabling accurate mental simulation.

This metric assesses local fidelity—how well the explanation captures the model's behavior for a specific instance. It is distinct from global interpretability methods that summarize overall model behavior. Key aspects include:

• It validates whether the explanation is faithful to the model's reasoning for that particular case.
• A successful explanation allows a human to 'step into the model's shoes' for that single decision.
• It is often used alongside automated faithfulness metrics (like infidelity or sufficiency) to provide a complementary, human-grounded assessment of local accuracy.

Simulatability occupies a unique niche in the explainability validation landscape:

• vs. Faithfulness/Completeness Scores: These are intrinsic, automated metrics. Simulatability provides an extrinsic, human-performance-based validation of those same properties.
• vs. Human-AI Agreement: Agreement measures if a human's reasoning aligns with the model's. Simulatability measures if a human can predict the model's output using the explanation.
• vs. Stability: Stability checks if explanations are consistent for similar inputs. Simulatability tests if a single explanation is useful for understanding a single prediction. It is a crucial bridge between algorithmic explanation quality and practical human usability.

Implementing a simulatability evaluation requires a structured experiment:

1. Sample Selection: Choose a representative set of model input-output pairs.
2. Explanation Generation: Produce explanations (using SHAP, LIME, counterfactuals, etc.) for each instance.
3. Human Task Design: Present the (input, explanation) pair and ask for a prediction of the model's class/probability/regression value.
4. Scoring: Calculate the simulatability score as the agreement rate (e.g., accuracy, mean squared error) between human predictions and true model outputs. Higher scores indicate more simulatable, and therefore more useful, explanations.

The effectiveness of simulatability testing varies with the explanation modality:

• Feature Attribution Maps (e.g., saliency maps): Users must mentally integrate highlighted features. Success depends heavily on the user's domain expertise.
• Counterfactual Explanations (e.g., 'If X had been Y, the output would be Z'): Often highly simulatable, as they provide a clear, contrastive causal narrative.
• Rule-based Explanations (e.g., Anchors): Provide explicit logical conditions, which can lead to very high simulatability if the rule is concise and precise.
• Concept-based Explanations (e.g., TCAV): Require the user to understand the defined concepts, adding a layer of abstraction that can impact simulatability.

While powerful, simulatability has key limitations:

• Resource Intensive: Requires recruiting and compensating human evaluators, making it less scalable than automated metrics.
• Expertise Dependency: Results can vary significantly based on the evaluators' familiarity with the domain and the model's task.
• Not a Direct Faithfulness Proof: A human could correctly predict the output using a flawed explanation if they impose their own correct reasoning. It is a measure of explanatory utility, not a pure guarantee of causal faithfulness.
• Baseline Requirement: Should be compared against a control condition (e.g., prediction accuracy with no explanation) to measure the explanation's added value.

A faithfulness score is a quantitative metric that measures how accurately an explanation reflects the true reasoning process or causal factors of the underlying model for a given prediction. Unlike simulatability, which tests a human's predictive ability, faithfulness directly evaluates the explanation's alignment with the model's internal mechanics.

• Core Principle: A faithful explanation should identify features the model actually used, not just correlated ones.
• Common Measurement: Perturb input features based on the explanation's importance scores. A faithful explanation will cause a correspondingly large change in the model's output when high-importance features are altered.
• Contrast with Simulatability: Faithfulness is an intrinsic, model-centric metric, while simulatability is an extrinsic, human-centric metric. An explanation can be faithful but not easily simulatable (if complex), and vice-versa.

A completeness score is a metric that evaluates whether an explanation accounts for all features or factors that contributed significantly to a model's prediction. It ensures the explanation is not missing critical components.

• Purpose: To prevent explanations from being misleadingly sparse or partial. For instance, a loan denial explanation citing only 'credit score' is incomplete if 'debt-to-income ratio' was equally decisive.
• Relation to Simulatability: An incomplete explanation will hinder simulatability, as a human cannot accurately predict the model's output if key decision factors are omitted from the explanation provided to them.
• Measurement: Often calculated by verifying that the sum of attribution scores for all features approximates the model's output deviation from a baseline.

A stability score measures the consistency of explanations generated for similar inputs or under small perturbations, assessing the robustness of the explanation method itself. Also referred to as explanation robustness.

• Why It Matters: An unstable explanation method produces vastly different feature attributions for two nearly identical inputs, undermining trust and making simulatability exercises unreliable.
• Example: A sentiment classifier should attribute importance to similar words (e.g., 'great', 'excellent') in two reviews both labeled 'positive'. High volatility in attributions indicates low stability.
• Impact on Simulatability: Low stability confounds the human evaluator in a simulatability test, as they cannot discern a consistent pattern from the explanations to inform their predictions.

Human-AI agreement is an extrinsic evaluation metric that measures the degree of alignment between a model's explanation and the reasoning or feature importance assigned by a human expert for the same prediction.

• Evaluation Method: Domain experts are shown an input and the model's prediction, then asked to generate their own explanation or rank feature importance. The correlation between the expert's ranking and the AI's explanation is calculated.
• Difference from Simulatability: Simulatability tests predictive performance using the explanation. Human-AI agreement tests explanatory content against expert judgment. They measure different aspects of explanation quality.
• Use Case: Critical in high-stakes domains like medicine or finance, where explanations must align with established domain logic to be trusted.

Contrastive explanations are a type of explanation that answers 'why P rather than Q?' by highlighting the features most responsible for the model choosing prediction P over a contrasting alternative Q.

• Structure: Focuses on the difference between the actual instance and a counterfactual one. For example, 'Your loan was approved (P) rather than denied (Q) because your income is above $X, even though your credit history is short.'
• Utility for Simulatability: Providing a contrastive explanation can significantly improve a human's simulatability score. By clarifying the decision boundary, it helps the user predict how the model would behave for small variations around the input.
• Link to Counterfactuals: Closely related to counterfactual explanations, but where counterfactuals provide a new input to change the outcome, contrastive explanations explain the divergence between two outcomes for the given input.

Local fidelity is a property of a post-hoc explanation that measures how well the explanation approximates the behavior of the complex model in the immediate vicinity of a specific input instance.

• Technical Definition: A locally faithful explanation acts as a surrogate model (e.g., a linear model) that closely matches the predictions of the black-box model for inputs near the instance being explained.
• Foundation for Simulatability: High local fidelity is a prerequisite for high simulatability. If the explanation is not faithful to the model's behavior locally, a human using it will fail to accurately predict outputs for the original input or similar ones.
• Methods Ensuring Fidelity: Techniques like LIME (Local Interpretable Model-agnostic Explanations) are explicitly designed to optimize local fidelity by sampling points around the instance and fitting an interpretable model.