Human-AI Agreement

Unlike intrinsic metrics that measure technical properties of the explanation itself, Human-AI agreement is an extrinsic metric. It evaluates the explanation's effectiveness for a downstream human-in-the-loop task, such as decision support, model debugging, or trust calibration. High agreement indicates the explanation successfully communicates the model's rationale in terms a human expert finds credible and actionable.

The metric's ground truth is derived from human judgment, not the model's internal weights. It typically involves:

• Expert Elicitation: Domain experts provide their own feature importance rankings or reasoning for a set of predictions.
• Alignment Scoring: A correlation measure (e.g., Spearman's rank, Kendall's Tau) is calculated between the expert's ranking and the model explanation's ranking.
• This process validates if the explanation faithfully represents the model's logic in a human-comprehensible way.

Agreement is not absolute; it varies based on critical context:

• Domain Expertise: Agreement scores differ between novices and domain experts.
• Explanation Format: Scores may vary for feature attributions (SHAP, LIME) versus counterfactual explanations or natural language rationales.
• Task Complexity: Agreement is harder to achieve on complex, multi-factorial predictions than on simpler ones.
• This necessitates careful experimental design when benchmarking explanation methods using this metric.

Human-AI agreement is distinct from, but complementary to, faithfulness scores. A perfectly faithful explanation (accurately reflecting the model's mechanics) may still have low human agreement if it is overly complex or references non-intuitive features. Conversely, a simple explanation with high human agreement might omit critical but obscure model factors, scoring low on faithfulness. Effective explainability requires balancing both metrics.

This metric is critical in high-stakes, human-collaborative domains:

• Healthcare Diagnostics: Do a model's saliency maps on a medical scan align with a radiologist's areas of concern?
• Financial Fraud Detection: Does the model's reason for flagging a transaction match an analyst's suspicion?
• Model Debugging & Auditing: Engineers use agreement to identify when a model relies on spurious correlations that experts would reject.
• It bridges the gap between model interpretability and real-world operational utility.

Common protocols for quantifying agreement include:

• Rank Correlation: Experts rank feature importance; correlation is computed with the explanation's ranking.
• Forced-Choice Selection: Present experts with multiple explanations; measure how often they select the model's explanation as matching their own reasoning.
• Simulatability Tasks: After seeing the input and explanation, can a human accurately predict the model's output? Success rate measures agreement.
• These methods move beyond abstract scores to concrete, task-based evaluation.

In medical diagnostics, a model's explanation for a predicted condition (e.g., a tumor malignancy score) is compared to a radiologist's annotated regions of interest. High Human-AI agreement validates that the model's saliency maps align with established medical knowledge, building clinician trust and supporting regulatory submissions for software as a medical device (SaMD). This is crucial for tools analyzing chest X-rays, retinal scans, or histopathology slides.

When a model flags a transaction as fraudulent, compliance officers must understand why. Human-AI agreement is measured by comparing the model's feature attribution (e.g., high importance on 'transaction velocity' or 'geographic mismatch') with an investigator's report. High agreement ensures the model's reasoning is auditable and justifiable, meeting requirements from regulations like the EU's AI Act and enabling faster, more reliable investigative workflows.

For credit scoring models, regulations like the Equal Credit Opportunity Act (ECOA) in the U.S. require adverse action notices that explain denials. Human-AI agreement validates that the model's top reasons for denial (e.g., high debt-to-income ratio, short credit history) match the reasons a human underwriter would cite. This process is part of post-hoc explanation validation to ensure algorithmic decisions are fair, transparent, and legally defensible.

Platforms use AI to flag hate speech or violent content. Human-AI agreement assesses whether the text spans or features the model identifies as problematic (its explanation) align with a human moderator's judgment. This metric is used to calibrate the model's sensitivity, reduce false positives, and provide clearer justifications for enforcement actions to users, which is critical for trust and safety operations at scale.

In fields like molecular biology or material science, models predict new drug candidates or stable compounds. Researchers use Human-AI agreement to check if a model's explanation for a prediction (e.g., highlighting a specific molecular substructure) corresponds to a domain expert's hypothesis. This agreement helps prioritize experimental validation, turning black-box predictions into credible, hypothesis-driven research leads.

When a model predicts a machine failure, maintenance engineers need to know which sensor readings (vibration, temperature) drove the alert. Human-AI agreement is calculated by comparing the model's feature importance scores with an engineer's diagnosis based on the same telemetry data. High agreement accelerates root cause analysis, ensures actionable alerts, and builds operator confidence in the AI system's recommendations.

A faithfulness score is a quantitative metric that measures how accurately an explanation reflects the true reasoning process or causal factors of the underlying model for a given prediction. It is a core intrinsic validation metric, distinct from human agreement.

• Key Idea: A faithful explanation correctly identifies the features the model actually used, not just features a human finds plausible.
• Measurement: Often calculated via perturbation analysis, where features deemed important by the explanation are removed or altered; a faithful explanation will correlate with a large drop in model prediction confidence.
• Contrast with Human-AI Agreement: Faithfulness is model-centric, while Human-AI Agreement is human-centric. An explanation can be faithful but not align with human intuition, and vice-versa.

Perturbation analysis is an explanation validation technique that systematically modifies or removes input features to observe the resulting changes in the model's output. It is the primary experimental method for calculating metrics like faithfulness and infidelity.

• Process: Features are perturbed (e.g., masked with zeros, replaced with baseline values) in order of importance as stated by an explanation. The subsequent drop in the model's prediction score is measured.
• Purpose: It tests the causal relationship between an explained feature and the model's output. A large performance drop upon removing a high-importance feature validates the explanation.
• Common Metrics: This technique directly computes Infidelity (the degree of failure) and is used to approximate Sufficiency and Completeness scores.

Infidelity is an explanation metric that quantifies the degree to which an explanation fails to accurately reflect the model's output when the input is perturbed according to the explanation's importance scores. It is a direct measure of unfaithfulness.

• Calculation: Given an importance score vector and a meaningful perturbation (e.g., blurring, noise), infidelity measures the expected squared error between the model's output change and the dot product of the importance scores and the perturbation.
• Interpretation: A low infidelity score indicates high faithfulness; the explanation reliably predicts how the model will behave when features are changed.
• Role in Validation: Serves as a crucial automated, model-grounded check that complements human-centric evaluations like Human-AI Agreement.

Sufficiency and Completeness are complementary metrics that evaluate whether an explanation captures the right amount of information about a model's prediction.

• Sufficiency: Measures whether the subset of features identified as most important by an explanation is, by itself, sufficient for the model to make its original prediction. A sufficient explanation means the top-K features yield a prediction score nearly identical to the full input.
• Completeness: Evaluates whether an explanation accounts for all features that contributed significantly to the prediction. A complete explanation's importance scores sum to the difference between the model's prediction for the actual input and a baseline input.
• Practical Use: Together, they help diagnose if an explanation is overly sparse (high sufficiency but low completeness) or overly dense (high completeness but low sufficiency).

Simulatability is an evaluation criterion for explanations that measures how well a human can use the provided explanation to accurately predict the model's output for a given input. It is a key bridge between intrinsic metrics and human understanding.

• Test Method: In a user study, participants are given an input and its corresponding explanation, then asked to predict the model's label or output value. The accuracy of their predictions is the simulatability score.
• Connection to Human-AI Agreement: While Human-AI Agreement measures alignment on feature importance, simulatability measures alignment on the final prediction outcome enabled by the explanation.
• Value: High simulatability indicates the explanation is effectively teaching the human the model's local behavior, a critical goal for trustworthy human-in-the-loop systems.

Explanation robustness refers to the property of an explanation method to produce consistent and stable attributions for a given prediction when the input or model is subjected to minor, semantically-preserving perturbations.

• Why It Matters: An explanation that changes drastically for imperceptible changes in the input (e.g., a single pixel shift in an image) is unreliable and cannot be trusted for debugging or compliance.
• Evaluation: Measured via a Stability Score, which quantifies the variance in explanation outputs across a set of perturbed versions of the same input instance.
• Relation to Validation: A robust explanation is a prerequisite for meaningful Human-AI Agreement studies; if the explanation itself is unstable, asking humans to agree with it becomes an ill-posed task.