Model Interpretability Score

The degree to which the explanation accurately reflects the model's true internal reasoning process. A high-fidelity score indicates that the interpretability method is not just a plausible story, but a faithful representation of the actual computation.

• Local Fidelity: Accuracy of the explanation for a single prediction.
• Global Fidelity: Accuracy of the explanation for the model's overall behavior.
• Contrast: A method with perfect fidelity on a linear model may have near-zero fidelity on a deep ensemble.

The consistency of an explanation under minor, semantically insignificant perturbations to the input. A stable score ensures that explanations do not fluctuate wildly, which would undermine user trust and audit reliability.

• Local Lipschitz Continuity: Measures how much an explanation changes relative to input change.
• Adversarial Stability: Tests whether an attacker can craft inputs that radically alter the explanation without changing the prediction.
• Practical Example: Adding a single pixel to an image should not completely invert a feature attribution map.

A measure of the cognitive load required for a human to grasp the explanation. This property directly addresses the 'human' element in human-in-the-loop oversight, ensuring the score reflects practical usability.

• Sparsity: The number of features or rules in the explanation. Fewer elements generally mean higher comprehensibility.
• Cognitive Chunking: Whether the explanation uses high-level, semantic concepts instead of raw input features (e.g., 'nose shape' vs. pixel 4,392).
• Metric: Often measured via user studies tracking task completion time and error rate.

The property ensuring that the relationship between an input feature and the model's output is directionally consistent and non-paradoxical within the explanation. This is critical for regulated domains like credit scoring.

• Constraint: If feature A increases the risk score, the explanation must never show feature A decreasing the risk score in a different context without a clear, documented interaction.
• Global vs. Local: A model can be locally monotonic for a specific decision while violating global monotonicity.
• Regulatory Alignment: Directly supports the 'right to explanation' by preventing illogical justifications.

The extent to which the explanation relies on causal drivers rather than mere statistical correlations. A high score here distinguishes between a model that has learned spurious shortcuts and one that has learned the true data-generating process.

• Confounder Robustness: The explanation should not attribute importance to a feature that is only correlated with the outcome through a hidden confounder.
• Interventionist: Ideally, the explanation should predict what would happen to the output if you physically intervened to change the input feature.
• Counterfactual Validity: A causally aligned explanation produces counterfactuals that respect real-world feasibility constraints.

The capacity for a human operator to use the explanation to achieve a specific goal, such as contesting a decision or debugging a model failure. This property transforms interpretability from a passive observation into an active governance tool.

• Recourse: Can the explanation guide a rejected loan applicant on exactly what to change to get approved?
• Debugging Utility: Can an engineer identify a data poisoning vector by inspecting the explanation?
• Threshold: An actionable score implies the explanation is both contrastive (why this and not that?) and selective (focusing only on the few factors that truly matter).

The mathematical techniques that power interpretability scores by assigning importance values to input features. SHAP (SHapley Additive exPlanations) uses cooperative game theory to guarantee consistent feature contribution, while LIME (Local Interpretable Model-agnostic Explanations) approximates the model locally with an interpretable surrogate. Integrated Gradients computes the path integral of gradients from a baseline to the input, satisfying the sensitivity axiom that SHAP and LIME may violate.

A complementary approach to scoring that answers 'what would need to change for a different outcome?' rather than 'which features mattered most?' A counterfactual generates the minimal set of input perturbations required to flip a prediction—for example, 'if your income were $5,000 higher, the loan would be approved.' This method is explicitly referenced in GDPR Recital 71 as a mechanism for meaningful explanation.

A fundamental distinction in how interpretability scores are derived. Intrinsic interpretability uses inherently transparent models like decision trees, logistic regression, or Generalized Additive Models (GAMs) where the score is a direct readout of learned parameters. Post-hoc interpretability applies explanation techniques to black-box models after training. The trade-off is accuracy vs. explainability—deep neural networks often outperform interpretable models but require post-hoc methods that may introduce approximation error into the score.

Defines the scope of what an interpretability score measures. Global interpretability quantifies overall model behavior—which features drive predictions across the entire dataset, often visualized through feature importance plots or partial dependence plots. Local interpretability explains a single prediction, answering 'why was this specific loan denied?' Local scores are critical for right to explanation compliance under GDPR Article 22, where each automated decision affecting an individual must be explainable.

Meta-metrics that evaluate the quality of the interpretability score itself. Faithfulness measures whether the explanation accurately reflects the model's true reasoning—tested by removing 'important' features and verifying prediction degradation. Stability assesses whether similar inputs produce similar explanations; a low stability score indicates the explanation method is brittle. These are essential for auditing because an unfaithful but plausible-looking explanation creates a false sense of transparency.

The structured artifact where interpretability scores are published for stakeholder consumption. A Model Card—standardized by Google Research—includes intended use, evaluation results, and interpretability analysis. Under the EU AI Act, high-risk systems require transparency documentation that includes explanation capabilities. The interpretability score becomes a key field in these documents, enabling procurement teams to compare vendors on explainability posture.