Causal Bandit

Unlike standard bandits that learn from correlations, causal bandits operate on a causal graph (a directed acyclic graph) that encodes domain knowledge about confounders, mediators, and colliders. This allows the algorithm to identify the true causal effect of an action, such as a discount, by adjusting for variables like seasonality or user intent that affect both the action selection and the reward.

Causal bandits use do-calculus to formally distinguish between observing an action and intervening to set an action. The do(X=x) operator simulates a randomized controlled trial from observational data. This prevents the algorithm from learning spurious associations, such as recommending umbrellas because it observes high sales during rain, and instead learns the causal effect of the recommendation itself.

The primary advantage over standard contextual bandits is robustness to unobserved confounders. By leveraging instrumental variables or front-door criteria, a causal bandit can recover unbiased policy gradients even when a hidden variable influences both the context and the reward. This is critical in dynamic pricing where competitor actions are often unobserved but confound the relationship between price and demand.

The optimization objective extends standard regret minimization to incorporate causal constraints. The algorithm minimizes cumulative regret relative to the optimal causal policy, not just the best observed policy. This involves solving a bi-level optimization problem where the inner loop estimates the conditional average treatment effect (CATE) and the outer loop selects the action that maximizes the causal uplift.

Standard off-policy estimators like Inverse Propensity Scoring (IPS) are biased under confounding. Causal bandits employ doubly robust estimators augmented with propensity score models that account for the causal graph structure. This allows for safe, unbiased evaluation of a new recommendation policy using logged data collected under a prior, potentially confounded, logging policy.

A common implementation is the uplift bandit, which models the incremental effect of an action over a baseline. The reward signal is redefined as the uplift: the difference between the outcome with treatment and the outcome without. This targets only persuadable users, avoiding wasted incentives on 'sure things' or 'lost causes', directly optimizing for return on intervention spend.

A predictive modeling technique that directly estimates the causal effect of an action on an individual's outcome. Unlike standard bandits that optimize for reward, uplift modeling segments users into four groups: Persuadables (respond only if treated), Sure Things (convert regardless), Lost Causes (never convert), and Sleeping Dogs (convert only if not treated). Causal bandits leverage uplift models to target only the Persuadables, avoiding wasted incentives on the other segments and preventing negative treatment effects.

A statistical method for estimating the performance of a new causal policy using historical data collected under a different logging policy. This is critical for causal bandits because you cannot observe the outcome of an action you did not take. Key techniques include:

• Inverse Propensity Scoring (IPS): Re-weights observed rewards by the inverse probability of the action being taken by the logging policy, correcting for selection bias.
• Doubly Robust Estimator: Combines IPS with a direct reward model to provide unbiased estimates even if one of the two models is misspecified.

The core challenge that distinguishes causal bandits from standard contextual bandits. A confounder is a variable that influences both the action selection and the outcome, creating a spurious correlation. For example, a retailer might show discounts to high-intent users, making the discount appear more effective than it truly is. Causal bandits use techniques like back-door adjustment, instrumental variables, and propensity score matching to isolate the true intervention effect from these confounding influences.

An extension of the Bayesian Thompson Sampling algorithm that incorporates causal structure. Instead of maintaining a posterior distribution over expected rewards, it maintains a posterior over causal parameters in a structural equation model. At each step, it samples a causal graph from the posterior and selects the action that maximizes the expected outcome under that sampled causal model. This naturally balances the exploration-exploitation trade-off while respecting the underlying causal relationships.

The process of assessing a target causal policy's value using data generated by a different behavior policy. This is essential for safely validating causal bandit models before production deployment. Unlike standard off-policy evaluation, causal off-policy methods must account for unobserved confounders and ensure that the logging policy provides sufficient coverage over the action space. Techniques like marginalized importance sampling are used to evaluate policies under unobserved confounding.

A variant where the underlying causal relationships change over time, requiring the algorithm to continuously adapt. For example, the causal effect of a promotional email on purchase likelihood may shift during holiday seasons or economic downturns. Solutions include:

• Sliding window approaches that discount old observations.
• Change-point detection to identify when the causal structure shifts.
• Meta-learning to quickly adapt to new causal regimes based on past experience.