Doubly Robust Estimator

The defining characteristic of the DR estimator is its double robustness: it yields an unbiased estimate of a new policy's value if either the propensity score model or the reward outcome model is correctly specified. It does not require both to be correct.

• Safety net: If the logging policy propensities are accurate, IPS corrects for any bias in the reward model.
• Variance reduction: If the reward model is accurate, it provides a low-variance baseline, and the IPS term corrects the residual bias.
• Misspecification tolerance: Only fails when both models are simultaneously wrong, which is a strictly weaker condition than either IPS or DM alone.

The DR estimator for a target policy π_e, given logged data D collected under behavior policy π_b, is:

V̂_DR = (1/n) Σ [ r̂(x_i, a_i) + (I(a_i = π_e(x_i)) / π_b(a_i | x_i)) · (r_i - r̂(x_i, a_i)) ]

• r̂(x, a): The predicted reward from the direct outcome model.
• π_b(a|x): The propensity of the logging policy.
• Correction term: The IPS weight is applied only to the residual (r - r̂), not the full reward, which dramatically reduces variance compared to pure IPS.

DR estimators achieve lower variance than pure IPS by using the direct model as a control variate. The variance depends on the accuracy of the reward model.

• Perfect reward model: If r̂(x, a) = E[r|x, a], the correction term has zero mean, and variance reduces to that of the DM estimator.
• Imperfect reward model: Variance scales with the residual errors, but is still weighted by inverse propensities, so extreme propensity weights can still inflate variance.
• Clipping propensities: In practice, propensities are often clipped to a minimum threshold (e.g., 0.01) to prevent variance explosion from near-zero probabilities.

DR sits at the intersection of two families of off-policy estimators, inheriting strengths from both:

• Inverse Propensity Scoring (IPS): Unbiased but high-variance, especially when behavior and target policies diverge significantly.
• Direct Method (DM): Low-variance but biased if the reward model is misspecified.
• Doubly Robust: Unbiased if either component is correct, with variance typically between DM and IPS.
• Switch-DR: An extension that dynamically chooses between DR and DM based on estimated variance to further optimize the bias-variance trade-off.

Deploying DR estimation in production bandit or recommendation systems requires careful engineering:

• Propensity logging: Every decision must log the probability of the chosen action under the logging policy. This is non-negotiable.
• Reward model training: The outcome model r̂(x, a) should be trained on a held-out dataset to avoid overfitting bias.
• Cross-fitting: Use K-fold cross-fitting where the reward model for each fold is trained on data not used for evaluation, eliminating bias from overfitting.
• Continuous monitoring: Track the effective sample size and variance of DR estimates to detect when the behavior policy has drifted too far from the target policy.

DR estimators are the workhorse of safe, offline policy evaluation in production:

• Contextual bandits: Evaluating a new exploration strategy before exposing real users to potentially suboptimal actions.
• Recommender systems: Comparing candidate ranking models using historical click logs without an A/B test.
• Dynamic pricing: Estimating revenue under a new pricing policy using data from the current pricing strategy.
• Clinical trials: Adapting treatment assignment policies while maintaining unbiased estimates of treatment effects.

A regression-based approach that trains a model to predict the reward for every possible action, then evaluates a target policy by summing the predicted rewards for the actions it would take.

• Strength: Low variance because it relies on smooth function approximation.
• Weakness: High bias if the reward model is misspecified or fails to capture complex interactions.
• Doubly Robust Logic: The estimator subtracts the DM's biased estimate for the chosen action and adds back the IPS-weighted residual, canceling the bias if the DM is wrong.

The broader statistical framework for answering 'what would have happened if we used a different policy?' using only historical logs.

• Goal: Safely estimate a new policy's performance without an expensive, risky A/B test.
• Key Assumption: Unconfoundedness—the logging policy's action choice must depend only on observed context features, not on hidden variables.
• Application: Doubly Robust is the gold-standard estimator for counterfactual evaluation in contextual bandits and recommender systems.

The conditional probability that the logging policy selected a specific action given the observed context.

• Logging Requirement: Propensities must be strictly greater than zero for all actions to satisfy the common support or positivity assumption.
• Clipping: In practice, extreme propensities near 0 or 1 are often clipped to a small epsilon to control the variance of IPS-based estimators.
• Storage: Production bandit systems must log the exact propensity alongside each decision to enable later off-policy evaluation.

The optimization objective that the Doubly Robust Estimator helps measure offline. Regret is the cumulative difference between the reward of the optimal policy and the reward actually accumulated.

• Off-Policy Role: By providing unbiased estimates of a new policy's value, Doubly Robust allows engineers to select the policy with the lowest expected regret before deployment.
• Model Selection: Comparing the Doubly Robust estimates of multiple candidate policies is a standard way to perform offline hyperparameter tuning for bandits.

The partial-information learning signal where only the reward for the chosen action is observed, leaving the outcomes of all other actions unknown.

• The Core Challenge: This missing counterfactual data is precisely what off-policy estimators must reconstruct.
• Doubly Robust Advantage: By combining a reward model that imputes missing outcomes with IPS that corrects the imputation, the estimator makes efficient use of the limited bandit feedback.
• Contrast: This differs from full-information supervised learning where the correct label is always known.