Doubly Robust Estimation

The DR estimator blends two approaches into a single equation:

• Direct Method (DM): Predicts the outcome for each action using a regression model.
• Inverse Propensity Scoring (IPS): Re-weights observed outcomes by the inverse probability of the action being taken by the logging policy.

The DR estimator corrects the DM's prediction by adding the IPS-weighted residual error. This structure mathematically guarantees that if the DM is wrong, the IPS correction fixes the bias, and vice versa.

The term 'doubly robust' refers to the estimator's unique consistency guarantee. An estimate is consistent if at least one of two nuisance models is correctly specified:

• Propensity Model: Correctly models the probability of the historical action.
• Outcome Model: Correctly predicts the expected reward for each action.

If both models are slightly misspecified, the bias is often smaller than using either method alone. This property is critical in causal inference and off-policy evaluation where perfect model specification is rare.

For a given context x, action a, observed reward r, and estimated propensity π(a|x), the DR estimator for the value of a target policy π_e is:

V_DR = 1/N Σ [ μ(x, a) + (r - μ(x, a)) * π_e(a|x) / π_b(a|x) ]

Where:

• μ(x, a) is the predicted outcome from the direct model.
• π_b(a|x) is the behavior policy's propensity.
• The term (r - μ(x, a)) is the residual.

If μ is perfect, the residual is zero, and the estimator equals the DM. If π_b is perfect, the IPS term provides an unbiased correction.

While DR estimation reduces bias, it introduces a specific variance profile:

• High Variance with Small Propensities: Like IPS, if the behavior policy assigns a very low probability to an action the target policy favors, the importance weight π_e/π_b can explode, causing high variance.
• Clipping Weights: A common regularization technique is to clip importance weights to a maximum threshold, trading a small amount of bias for a significant reduction in variance.
• Self-Normalization: Applying a self-normalizing step to the weights can stabilize estimates in practice.

In a Next-Best-Action (NBA) system, DR estimation is used to evaluate a new candidate policy offline before deployment:

1. Collect Logs: Gather historical data from the current production policy (e.g., random offers or a previous model).
2. Train Nuisance Models: Build a propensity model for the logging policy and an outcome model to predict customer response.
3. Evaluate Candidate Policy: Use the DR estimator to calculate the expected reward of the new NBA policy.
4. Validate Safely: Only deploy the new policy if the DR estimate shows a statistically significant improvement over the current baseline.

DR estimation sits in a spectrum of Off-Policy Evaluation (OPE) techniques:

• Inverse Propensity Scoring (IPS): Unbiased but suffers from extremely high variance when propensities are small. Requires no outcome model.
• Direct Method (DM): Low variance but suffers from high bias if the outcome model is misspecified. Requires no propensity model.
• Doubly Robust (DR): Aims for the best of both. It is unbiased if the propensity model is correct, and often has lower variance than pure IPS if the outcome model is a reasonable predictor.
• Self-Normalized IPS (SNIPS): Reduces IPS variance but introduces a small bias; can be combined with DR.

An off-policy evaluation method that corrects for selection bias in logged data by re-weighting observed outcomes by the inverse probability of the action being taken by the logging policy. IPS is one of the two foundational components of doubly robust estimation.

• Formula: V_IPS = (1/n) * Σ [ (π_e(a|x) / π_b(a|x)) * r ]
• Suffers from high variance when propensity weights are extreme
• Provides unbiased estimates when the propensity model is correctly specified
• Forms the propensity score half of the doubly robust estimator

A regression-based approach that trains a model to directly predict the expected reward for each possible action given a context, then evaluates a policy by averaging these predictions. This is the second component of doubly robust estimation.

• Also called the regression estimator or reward model
• Low variance but susceptible to model misspecification bias
• Uses supervised learning to estimate E[r | x, a]
• Doubly robust estimation combines this with IPS to achieve double protection against misspecification

A set of statistical techniques used to estimate the performance of a new target policy using historical data collected by a different, often suboptimal, behavior policy. Doubly robust estimation is one of the most reliable OPE methods.

• Critical for safely evaluating NBA models before production deployment
• Avoids costly and risky online A/B tests
• Other OPE methods include: Direct Method, IPS, and Doubly Robust
• The fundamental challenge is correcting for the distribution shift between logging and target policies

The process of drawing conclusions about cause-and-effect relationships from data, moving beyond correlation to determine the true impact of an intervention. Doubly robust estimation is a core tool in the causal inference toolkit.

• Addresses the fundamental problem of causal inference: we can never observe the counterfactual outcome
• Relies on assumptions like unconfoundedness and positivity
• Used to estimate treatment effects in marketing, healthcare, and policy
• Doubly robust methods provide consistent estimates under weaker assumptions than single-model approaches

The variation in the causal effect of an intervention across different individuals or subgroups in a population. Doubly robust estimation enables accurate HTE estimation by combining propensity and outcome models.

• Indicates that a single treatment does not have a uniform impact on all users
• Critical for personalized next-best-action decisions
• Estimated using Conditional Average Treatment Effect (CATE) models
• Doubly robust learners like DR-Learner and R-Learner are state-of-the-art for HTE estimation

A statistical technique that estimates the probability of a customer receiving a specific treatment given their observed characteristics. Accurate propensity models are essential for the IPS component of doubly robust estimation.

• Typically estimated using logistic regression or gradient-boosted trees
• Must satisfy the positivity assumption: 0 < P(treatment|x) < 1 for all x
• Poorly calibrated propensity scores lead to biased doubly robust estimates
• Often combined with calibration techniques like Platt scaling to improve accuracy