Causal Reinforcement Learning

Unlike standard model-based RL that learns a predictive model P(next state | current state, action), Causal RL learns an interventional model P(next state | do(action), current state). This model answers 'what if' questions, allowing the agent to simulate the effects of actions without executing them. This is crucial for sample-efficient planning and evaluating counterfactual policies.

• Key Mechanism: Uses the do-operator from causal calculus to sever incoming edges to the action node in the learned causal graph.
• Benefit: Enables robust planning under distribution shifts, as the causal relationships are more stable than correlational patterns.

A core feature is the agent's ability to autonomously discover the causal graph of its environment from interaction data. This involves identifying which variables cause others, distinguishing direct from indirect effects, and detecting confounders.

• Methods: Algorithms combine RL exploration with conditional independence tests (e.g., PC, FCI algorithms) or score-based structure learning.
• Outcome: The learned graph acts as a skeleton for generalization. The agent understands which environment dynamics will remain invariant if parts of the system change, leading to more robust policies in novel situations.

Causal RL seeks policies that are invariant to spurious correlations and non-causal associations. It focuses on learning from causal features—those that have a genuine mechanistic effect on rewards—rather than all observable features.

• Approach: Inspired by Invariant Risk Minimization (IRM), the agent is trained across multiple, diverse environments or contexts. The policy is optimized to perform well in all contexts by leveraging only the invariant, causal relationships.
• Result: Dramatically improved out-of-distribution (OOD) generalization. The agent avoids strategies that work by accident in training but fail when superficial correlations break.

This feature enables the agent to learn from counterfactual outcomes—what the reward would have been had a different action been taken. This is a more powerful learning signal than the observed reward alone.

• Process: The agent uses its causal model to estimate the expected value of actions it did not take, given the current state. Regret is calculated as the difference between the value of the best counterfactual action and the value of the action taken.
• Impact: Leads to faster credit assignment in long-horizon tasks and more data-efficient learning, as each real trajectory provides implicit information about many alternative paths.

Causal RL agents perform targeted exploration to resolve causal uncertainty. Instead of exploring randomly or for pure reward discovery, they design interventions to test specific causal hypotheses.

• Strategy: The agent identifies ambiguous edges or confounded relationships in its partial causal graph. It then chooses actions that act as controlled experiments to disentangle these relationships (e.g., intervening on a variable while holding others fixed).
• Advantage: This active causal learning reduces the number of episodes needed to learn an accurate world model, achieving superior sample efficiency compared to curiosity-driven or random exploration.

Causal RL facilitates knowledge transfer between tasks with different low-level observations but shared high-level causal mechanics. The agent learns abstract causal representations that are portable across domains.

• Mechanism: The agent identifies causal equivalence classes. For example, the abstract rule 'increasing force causes greater acceleration' can be transferred from a simulated robot to a real one, even if sensor readings differ.
• Use Case: Enables sim-to-real transfer and cross-domain policy reuse by aligning the causal graphs of the source and target environments, then mapping policies at the abstract causal level.

A Structural Causal Model (SCM) is the foundational mathematical framework for causal reasoning. It represents causal relationships between variables using a system of structural equations, typically visualized as a causal graph (a Directed Acyclic Graph).

• Core Components: Each variable is defined by a function of its direct causes (parents in the graph) and an independent noise term.
• Enables Intervention: The do-operator is formally defined within an SCM, allowing precise modeling of interventions (do(X=x)).
• Basis for Counterfactuals: By specifying the equations and noise distributions, SCMs allow computation of counterfactual queries ("What would have happened if...?").

In Causal RL, an agent may learn or be provided with an SCM of its environment to predict the effects of its actions and plan more effectively.

Causal discovery refers to algorithms that automatically infer the causal structure of a system from data. Instead of assuming a known graph (as in standard causal inference), these methods attempt to learn it.

• Constraint-Based: Algorithms like PC and FCI test for conditional independencies in observational data to infer the graph.
• Score-Based: Methods search over graph structures to optimize a score (e.g., BIC) that trades off model fit and complexity.
• Challenges: Results are often a set of Markov equivalent graphs unless interventional data or specific assumptions (like non-linearities) are used.

For a Causal RL agent, causal discovery can be a sub-task—learning the environment's causal dynamics from its interaction experience to build a more accurate world model.

Do-calculus is a set of three inference rules developed by Judea Pearl that allows researchers to determine if and how a causal effect can be estimated from available data. It manipulates probability expressions involving the do-operator.

• Purpose: To transform an interventional probability P(Y | do(X)) into an expression containing only observational probabilities P(...), given a known causal graph.
• Enables Identification: If such a transformation is possible, the causal effect is identifiable from observational studies.
• Basis for Criteria: The backdoor and frontdoor criteria are practical graphical applications derived from do-calculus logic.

In Causal RL, do-calculus provides the formal machinery for an agent to reason about which observational data can be used to predict the outcome of a novel intervention (action).

Invariant Risk Minimization (IRM) is a learning paradigm designed to find predictors that generalize across diverse environments by discovering causal invariants. It addresses the problem of distribution shift, which is a key motivation for Causal RL.

• Core Idea: Learn a data representation such that the optimal classifier on top of that representation is the same across all training environments.
• Seeks Causal Features: Non-spurious, causal features should satisfy this invariance, while spurious correlations will vary.
• Connection to Causality: The framework formalizes the idea that causal mechanisms are stable, while associational patterns may change.

Causal RL agents aim for similar robustness. IRM's principles can inform how an RL agent learns state representations that capture causal, intervention-stable relationships, leading to better performance in new settings.

Model-Based Reinforcement Learning (MBRL) is a class of RL where the agent learns (or is given) an internal model of the environment's dynamics—the transition function T(s' | s, a) and reward function R(s, a). This model is then used for planning.

• Sample Efficiency: By planning with a model, MBRL can often require fewer interactions with the real environment.
• Key Distinction from Causal RL: A standard dynamics model predicts associations (next state given current state and action). A causal model within MBRL distinguishes between mere correlation and true causal effect, understanding the mechanisms behind transitions.
• Synergy: Causal RL often implements MBRL with a causal world model. This allows for more accurate predictions under interventions and better generalization to novel states or actions.

A counterfactual is the highest level of causal reasoning on Pearl's "Ladder of Causation." It answers "What would have happened if...?" questions about past events, considering what did happen.

• Formal Definition: Given observed evidence E=e, a counterfactual queries the probability of outcome Y had X been set to x', written as P(Y_{X=x'} | E=e).
• Requires a Full SCM: Computing counterfactuals requires knowledge of the structural equations and the specific noise values for the observed instance.
• Role in RL: In Causal RL, counterfactual reasoning allows an agent to learn from past trials more efficiently. For example: "Given that I took action A and failed, would I have succeeded if I had taken action B instead?" This enables robust credit assignment and policy improvement.