Causal Reinforcement Learning

Causal RL seeks policies that are invariant to spurious correlations and non-causal associations. It focuses on learning from causal features—those that have a genuine mechanistic effect on rewards—rather than all observable features.

• Approach: Inspired by Invariant Risk Minimization (IRM), the agent is trained across multiple, diverse environments or contexts. The policy is optimized to perform well in all contexts by leveraging only the invariant, causal relationships.
• Result: Dramatically improved out-of-distribution (OOD) generalization. The agent avoids strategies that work by accident in training but fail when superficial correlations break.

This feature enables the agent to learn from counterfactual outcomes—what the reward would have been had a different action been taken. This is a more powerful learning signal than the observed reward alone.

• Process: The agent uses its causal model to estimate the expected value of actions it did not take, given the current state. Regret is calculated as the difference between the value of the best counterfactual action and the value of the action taken.
• Impact: Leads to faster credit assignment in long-horizon tasks and more data-efficient learning, as each real trajectory provides implicit information about many alternative paths.

Causal RL agents perform targeted exploration to resolve causal uncertainty. Instead of exploring randomly or for pure reward discovery, they design interventions to test specific causal hypotheses.

• Strategy: The agent identifies ambiguous edges or confounded relationships in its partial causal graph. It then chooses actions that act as controlled experiments to disentangle these relationships (e.g., intervening on a variable while holding others fixed).
• Advantage: This active causal learning reduces the number of episodes needed to learn an accurate world model, achieving superior sample efficiency compared to curiosity-driven or random exploration.

Causal RL facilitates knowledge transfer between tasks with different low-level observations but shared high-level causal mechanics. The agent learns abstract causal representations that are portable across domains.

• Mechanism: The agent identifies causal equivalence classes. For example, the abstract rule 'increasing force causes greater acceleration' can be transferred from a simulated robot to a real one, even if sensor readings differ.
• Use Case: Enables sim-to-real transfer and cross-domain policy reuse by aligning the causal graphs of the source and target environments, then mapping policies at the abstract causal level.

Do-calculus is a set of three inference rules developed by Judea Pearl that allows researchers to determine if and how a causal effect can be estimated from available data. It manipulates probability expressions involving the do-operator.

• Purpose: To transform an interventional probability P(Y | do(X)) into an expression containing only observational probabilities P(...), given a known causal graph.
• Enables Identification: If such a transformation is possible, the causal effect is identifiable from observational studies.
• Basis for Criteria: The backdoor and frontdoor criteria are practical graphical applications derived from do-calculus logic.

In Causal RL, do-calculus provides the formal machinery for an agent to reason about which observational data can be used to predict the outcome of a novel intervention (action).

Invariant Risk Minimization (IRM) is a learning paradigm designed to find predictors that generalize across diverse environments by discovering causal invariants. It addresses the problem of distribution shift, which is a key motivation for Causal RL.

• Core Idea: Learn a data representation such that the optimal classifier on top of that representation is the same across all training environments.
• Seeks Causal Features: Non-spurious, causal features should satisfy this invariance, while spurious correlations will vary.
• Connection to Causality: The framework formalizes the idea that causal mechanisms are stable, while associational patterns may change.

Causal RL agents aim for similar robustness. IRM's principles can inform how an RL agent learns state representations that capture causal, intervention-stable relationships, leading to better performance in new settings.

Model-Based Reinforcement Learning (MBRL) is a class of RL where the agent learns (or is given) an internal model of the environment's dynamics—the transition function T(s' | s, a) and reward function R(s, a). This model is then used for planning.

• Sample Efficiency: By planning with a model, MBRL can often require fewer interactions with the real environment.
• Key Distinction from Causal RL: A standard dynamics model predicts associations (next state given current state and action). A causal model within MBRL distinguishes between mere correlation and true causal effect, understanding the mechanisms behind transitions.
• Synergy: Causal RL often implements MBRL with a causal world model. This allows for more accurate predictions under interventions and better generalization to novel states or actions.

A counterfactual is the highest level of causal reasoning on Pearl's "Ladder of Causation." It answers "What would have happened if...?" questions about past events, considering what did happen.

• Formal Definition: Given observed evidence E=e, a counterfactual queries the probability of outcome Y had X been set to x', written as P(Y_{X=x'} | E=e).
• Requires a Full SCM: Computing counterfactuals requires knowledge of the structural equations and the specific noise values for the observed instance.
• Role in RL: In Causal RL, counterfactual reasoning allows an agent to learn from past trials more efficiently. For example: "Given that I took action A and failed, would I have succeeded if I had taken action B instead?" This enables robust credit assignment and policy improvement.