Causal Policy Analysis

Represents the environment's dynamics as a directed acyclic graph (DAG) where nodes are variables and edges are direct causal links. An SCM defines a joint distribution over state variables, actions, and rewards using structural equations.

• Nodes: States (S), Actions (A), Rewards (R), Confounders (C)
• Edges: Direct causal influence (e.g., A → S')
• Equations: S' = f(S, A, U) where U is exogenous noise
• Key Insight: Encodes the data-generating process, not just observational correlations.

The mathematical formalization of setting a variable to a specific value, independent of its usual causes. P(R | do(A=a)) represents the reward distribution when we force the agent to take action a, cutting off all incoming edges to the action node.

• Observational: P(R | A=a) — passive correlation
• Interventional: P(R | do(A=a)) — active causation
• Mechanism: Replaces the structural equation for A with a constant
• Goal: Answers 'What will happen if we change the policy?' rather than 'What did happen?'

Evaluates retrospective 'what if' scenarios at the level of a single trajectory. Given an observed outcome, counterfactual analysis computes what the outcome would have been had the agent taken a different action in the same exact situation.

• Three-Step Process:
  1. Abduction: Infer the latent noise variables (U) from the observed fact
  2. Action: Apply the do() operator to set the alternative action
  3. Prediction: Compute the resulting outcome using the modified SCM
• Use Case: Explaining a specific crash by showing that any other action would have also failed due to an unobserved obstacle.

Identifies and neutralizes confounders—hidden variables that influence both the agent's action selection and the resulting reward, creating a spurious correlation. The backdoor criterion provides a graphical test to determine which variables must be controlled for to isolate the true causal effect.

• Confounder Example: A shared weather variable (C) causes both a specific action (A) and a high reward (R), making the action look deceptively good.
• Adjustment Formula: P(R | do(A)) = Σ_C P(R | A, C) * P(C)
• Goal: Debiases the policy evaluation to reveal the true effect of the action.

A technique used when unobserved confounding cannot be directly measured or adjusted for. An instrument is a variable (Z) that influences the action (A) but has no direct effect on the reward (R) and is independent of the unobserved confounders.

• Conditions for a Valid IV:
  • Relevance: Z must causally influence A
  • Exclusion: Z affects R only through A
  • Exogeneity: Z is independent of the confounders
• Application: Using a randomized recommendation (Z) to measure the true causal effect of a user's actual choice (A) on satisfaction (R).

The primary diagnostic goal of causal policy analysis. A policy exploits a spurious correlation if its high reward depends on a non-causal statistical association that will vanish under a distribution shift. Causal analysis systematically tests for this fragility.

• Test: Does the policy's performance hold under an intervention do(S=s) that breaks the correlation?
• Example: An autonomous vehicle policy that learns to brake based on the presence of a shadow rather than a pedestrian. Causal analysis reveals the shadow is a confounded, non-causal feature.
• Outcome: A causally robust policy that relies on invariant, mechanistic relationships.

A family of off-policy evaluation techniques that estimate how a new policy would perform using only historical data, without deploying it. This provides a causal explanation of potential outcomes by answering: 'What would have happened if we used a different policy?'

• Inverse Propensity Scoring (IPS): Reweights historical actions by the ratio of target to behavior policy probabilities
• Doubly Robust Estimation: Combines IPS with a reward model for lower variance
• Directly explains the causal effect of switching policies before risking production deployment

A causal interpretability method that systematically removes or occludes input features to measure the resulting change in policy output. By observing how performance degrades when a feature is withheld, you determine its causal necessity for the decision.

• Zero ablation: Replace feature with zero or mean value
• Permutation ablation: Shuffle feature values to break statistical dependencies
• Measures true causal influence, not just correlation, distinguishing spurious features from genuine drivers

A technique for inferring the underlying reward function that an expert is implicitly optimizing. Rather than learning a policy from a known reward, IRL observes expert demonstrations and reverse-engineers the latent objective that explains the behavior.

• Maximum Entropy IRL: Assumes expert is noisily optimal, yielding a probabilistic reward model
• Generative Adversarial Imitation Learning (GAIL): Uses adversarial training to match state-action distributions
• Explains intent by surfacing what the agent truly values, not just what it does

A game-theoretic solution concept adapted to reinforcement learning for fairly distributing credit among cooperating agents. The Shapley value computes each agent's marginal contribution by averaging its impact across all possible coalitions of other agents.

• Shapley Q-Value: Applies Shapley decomposition to the joint Q-function
• Difference Rewards: A computationally cheaper approximation that compares global vs. counterfactual reward
• Guarantees fairness axioms: Efficiency, symmetry, and additivity in credit assignment

An internal generative model of the environment learned by an agent, which can be probed and visualized to understand the agent's beliefs about state transitions. By inspecting the world model, you can verify whether the agent has learned a causally accurate mental model.

• Latent state rollouts: Generate imagined trajectories to see what the agent expects
• Disentanglement analysis: Check if latent dimensions correspond to meaningful causal factors
• Reveals hallucinations: Identify where the agent's internal model diverges from true environment dynamics

An explanation format that answers 'Why action A instead of action B?' by highlighting the minimal state differences that caused the policy to diverge. This is inherently causal because it identifies the specific features that, if changed, would alter the decision.

• Minimal sufficient explanation: Smallest set of features that justify the action choice
• Contrastive counterfactuals: 'If feature X had value Y instead, action B would be chosen'
• Aligns with human psychology: People naturally seek contrastive 'why this, not that?' explanations