Glossary

Interventional SHAP

A causal formulation of SHAP that computes feature attributions by breaking correlations through sampling from the marginal distribution, reflecting model behavior under intervention.

Get in touch Learn more

Data engineer managing feature store on laptop, feature definitions visible, casual data engineering session.

CAUSAL FEATURE ATTRIBUTION

What is Interventional SHAP?

Interventional SHAP is a causal formulation of SHAP that breaks feature correlations by sampling from the marginal distribution, estimating how a model's prediction changes when a feature is actively set to a specific value.

Interventional SHAP computes Shapley values by explicitly breaking the statistical dependence between features. Unlike Observational SHAP, which conditions on correlated features, this method samples a feature's value from its marginal distribution—effectively simulating a physical intervention. This approach answers the question: 'How would the prediction change if we forcibly set this feature to a specific value, ignoring its natural correlations?'

The resulting attributions reflect the model's behavior under an interventional distribution rather than the observational data manifold. This makes Interventional SHAP particularly valuable for debugging model logic and assessing causal sensitivity, as it reveals how the model truly uses a feature in isolation. However, it can evaluate the model on unrealistic, off-manifold data points when features are highly correlated in reality.

INTERVENTIONAL SHAP EXPLAINED

Frequently Asked Questions

Clear answers to the most common questions about the causal formulation of SHAP that breaks feature correlations by sampling from the marginal distribution.

Interventional SHAP is a causal formulation of SHAP that computes feature attributions by breaking the statistical dependence between features, sampling missing features from their marginal distribution rather than conditioning on observed values. Unlike Observational SHAP, which preserves feature correlations by conditioning on known features, Interventional SHAP answers the counterfactual question: 'How would the prediction change if we intervened to set this feature to a specific value, while leaving all other features at their population-level distribution?' This approach reflects the model's behavior under an external intervention, making it the correct choice when you need to understand causal mechanisms or when features are known to be causally independent. The key trade-off is that Interventional SHAP may evaluate the model on unrealistic data points that lie off the natural data manifold, but it provides a true causal decomposition that satisfies the consistency and efficiency axioms without requiring a causal graph.

Enabling Efficiency, Speed & Accuracy

Intelligent Analysis, Decision & Execution

We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.

Talk to Us

Search across company data

Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.

Useful when people spend too long searching or get different answers from different systems.

Enterprise searchRAGPermissions

Automate internal workflows

Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.

Useful when repetitive work moves across multiple tools and teams.

AI agentsWorkflow automationGovernance

Add AI to products and internal tools

Build assistants, guided actions, or decision support into the software your team or customers already use.

Useful when AI needs to be part of the product, not a separate tool.

AI integrationDecision supportModel routing

CAUSAL VS. CORRELATIONAL EXPLANATIONS

Interventional vs. Observational SHAP

A comparison of the two primary SHAP formulations for handling feature dependence, contrasting their sampling strategies, theoretical guarantees, and practical implications for model auditing.

Feature	Interventional SHAP	Observational SHAP
Sampling Strategy	Breaks correlations by sampling from the marginal distribution P(X_j)	Preserves correlations by conditioning on observed values P(X_j \| X_C)
Theoretical Foundation	Causal inference (Pearl's do-calculus)	Statistical conditioning (conditional expectation)
Feature Independence Assumption	Enforced by design during imputation	Not required; respects natural data manifold
Model Evaluation Context	Evaluates model on off-manifold data points	Evaluates model only on the observed data manifold
Axiomatic Compliance	Satisfies all original Shapley axioms including consistency	Violates consistency axiom when features are correlated
Sensitivity to Correlation	Attribution is insensitive to feature correlation structure	Attribution splits credit among correlated features
Computational Complexity	Lower; requires sampling from marginal distributions	Higher; requires estimating conditional expectations
Use Case	Auditing causal model behavior and regulatory compliance	Explaining predictions within natural data distribution

CAUSAL EXPLANATIONS

Key Properties of Interventional SHAP

Interventional SHAP breaks feature dependencies by sampling from the marginal distribution, providing a causal interpretation that reflects how the model behaves when a feature is actively manipulated rather than passively observed.

Causal Interpretation

Interventional SHAP answers the question: 'What happens if I set this feature to a specific value?' Unlike observational approaches that condition on correlated features, this method simulates an external intervention by breaking the natural statistical links between features.

Estimates the controlled direct effect of a feature
Reflects the model's behavior under active manipulation
Essential for decision support systems where actions change feature values

Marginal Distribution Sampling

The defining mechanism of interventional SHAP is sampling from the marginal distribution P(X_j) rather than the conditional distribution P(X_j | X_S). This deliberately ignores correlations present in the training data.

Replaces feature values with random draws from the background dataset
Breaks collinearity between the target feature and its correlates
Produces explanations that are independent of data-generating process assumptions

Efficiency Property Guarantee

Interventional SHAP preserves the efficiency axiom of Shapley values: the sum of all feature attributions exactly equals the difference between the model's prediction and the expected baseline value.

Local accuracy: Explanation matches the model output precisely
Enables budgeting of feature contributions
Provides a complete decomposition of the prediction with no unexplained residual

Observational vs. Interventional Trade-off

The choice between observational and interventional SHAP represents a fundamental trade-off in explainability:

Observational SHAP: Preserves feature correlations, reflects the natural data manifold, but may attribute importance to correlated features that are not causally relevant
Interventional SHAP: Breaks correlations, provides causal clarity, but may evaluate the model on unrealistic data points outside the training distribution
Select based on whether the use case requires passive explanation or active intervention guidance

Background Dataset Selection

The background dataset serves as the reference population for computing expected values and sampling marginal distributions. Its composition critically impacts the resulting explanations.

Should represent the baseline state of the world
Larger datasets provide more stable estimates but increase computation
Using the full training set as background yields global marginal expectations
Domain-specific subsets enable contextualized explanations for specific subpopulations

Computational Considerations

Interventional SHAP requires repeated model evaluations on synthetically constructed inputs where feature values are randomly permuted. This introduces computational overhead compared to observational methods.

Complexity scales with number of features and background samples
TreeSHAP provides exact interventional values for tree models in polynomial time
For deep learning models, DeepSHAP offers efficient approximations
Sampling strategies can reduce variance and accelerate convergence

About the author

Prasad Kumkar

CEO & MD, Inference Systems

Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.

His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.

Limited slotsGet a Free AI Consultation

How We Work

Custom AI workflows for your Business

One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.