Simulatability

The core test of simulatability requires a human to act as a surrogate model. Given an input and the model's explanation, the human must predict the model's output. High simulatability means the human's prediction matches the model's actual output with high accuracy. This is distinct from plausibility—an explanation can sound reasonable but fail to enable accurate prediction, revealing a gap between perceived and actual understanding.

Simulatability is bounded by human working memory constraints. An explanation is only simulatable if a person can hold its logic in mind without external aids. Key factors include:

• Rule count: Fewer than 7±2 decision rules
• Interaction depth: Minimal feature interactions
• Linearity: Additive models are more simulatable than those with complex non-linearities A 10,000-tree gradient-boosted ensemble has near-zero simulatability regardless of explanation quality.

Simulatability operates at two scales:

• Local simulatability: Can a human predict the model's output for a single, specific input after seeing its explanation? This is the most common evaluation setting.
• Global simulatability: Can a human predict the model's output for any arbitrary input from the domain? This requires the explanation to capture the model's complete decision boundary.

Most interpretability research targets local simulatability, as global simulatability is achievable only for inherently transparent models like linear regression or shallow decision trees.

Empirical evaluation follows a structured protocol:

1. Training phase: Human subjects study model explanations on a set of instances with known outputs.
2. Prediction phase: Subjects predict outputs for novel, unseen instances using only the explanation.
3. Metric: Forward simulation accuracy—the agreement rate between human predictions and model outputs.

Controls include measuring baseline human accuracy without explanations and comparing against the Bayes optimal predictor. Studies consistently show that even faithful explanations yield imperfect simulatability due to cognitive limitations.

These two concepts are orthogonal and often confused:

• Faithfulness: Does the explanation accurately describe the model's true computation?
• Simulatability: Can a human use the explanation to predict outputs?

A perfectly faithful explanation of a deep neural network may still have zero simulatability because the computation is too complex for human cognition. Conversely, a simple but unfaithful explanation can yield high simulatability—the human predicts well, but for the wrong reasons. The ideal explanation maximizes both dimensions.

Maximizing simulatability requires deliberate explanation design:

• Sparse feature selection: Highlight only the 2-5 most influential inputs
• Additive decomposition: Present contributions as a sum of independent effects
• Contrastive framing: Explain why output A rather than output B
• Rule-based summaries: Use if-then logic rather than weight matrices
• Avoid interaction terms: Suppress feature interactions unless they are the primary driver

These principles align with findings from cognitive psychology on how humans construct mental models of causal systems.

The degree to which a generated rationale accurately mirrors the true computational logic used by the model. A faithful explanation enables simulatability by exposing real decision boundaries, not post-hoc fabrications.

• Key Distinction: Faithfulness vs. Plausibility
• Measurement: Input erasure, sensitivity analysis
• Failure Mode: Plausible but unfaithful rationales mislead auditors

The direct experimental method for measuring simulatability. A human is given an explanation and a new, unseen input, then asked to predict the model's output. Accuracy of these predictions quantifies simulatability.

• Protocol: Human-subject experiments with held-out data
• Metric: Agreement rate between human prediction and model output
• Goal: Validate that explanations transfer to novel cases

Natural language descriptions of the minimal changes to an input that would alter the prediction. These directly support simulatability by teaching users the model's decision boundary through contrast.

• Example: 'Loan would be approved if debt-to-income ratio were below 36%'
• Mechanism: Identifies necessary and sufficient conditions
• Relationship: Complements feature attribution with actionable insight

The practice of reverse-engineering the internal algorithms and circuits within a neural network's weights. Full mechanistic understanding represents the theoretical upper bound of simulatability.

• Scope: Decomposing attention heads, MLP layers, and residual streams
• Output: Causal graphs of model computation
• Contrast: Mechanistic interpretability seeks ground-truth algorithms, not just behavioral predictions

Eliciting and inspecting the intermediate reasoning steps generated by large language models. When faithful, these traces allow a human to step through the model's logic and anticipate subsequent outputs.

• Technique: Few-shot prompting with explicit reasoning exemplars
• Risk: Chain-of-thought may be a plausible rationalization, not true cognition
• Validation: Requires independent simulatability testing

A technique that approximates complex model predictions locally with interpretable surrogate models. By exposing the local decision surface, LIME enables simulatability within a defined neighborhood of the input.

• Method: Perturb inputs, fit sparse linear model
• Output: Feature weights for a single prediction
• Limitation: Local fidelity does not guarantee global simulatability