Counterfactual Evaluation

Counterfactual evaluation addresses the core challenge that we can only observe the outcome of the action that was taken, not the outcomes of actions that were not taken. The logged data contains the factual outcome for the chosen action, but the counterfactual outcomes for all other possible actions remain unobserved. This missing data problem is the central obstacle that off-policy estimators must overcome to provide unbiased performance estimates.

IPS is the foundational off-policy estimator that corrects for selection bias by re-weighting each observed reward by the inverse of the probability that the logging policy chose that action. The core formula is:

• Estimator: ( V_{IPS} = \frac{1}{N} \sum_{i=1}^{N} \frac{\pi_e(a_i|x_i)}{\pi_b(a_i|x_i)} r_i )
• Key Requirement: The logging policy (\pi_b) must have a non-zero probability of selecting any action the target policy (\pi_e) might select (common support)
• Variance Challenge: IPS can suffer from high variance when the logging and target policies diverge significantly, leading to large importance weights

The doubly robust estimator combines IPS with a direct method reward model to provide a more reliable estimate. It remains unbiased if either the propensity model or the reward model is correctly specified, providing a crucial safety net:

• Direct Method Component: Learns a model (\hat{r}(x, a)) to predict the reward for any context-action pair
• Correction Term: Adds an IPS-weighted correction for the residuals of the direct method predictions
• Practical Advantage: In real-world applications where perfect propensity or reward models are rare, the doubly robust estimator often exhibits significantly lower variance than pure IPS

For off-policy evaluation to be valid, the logging policy must have explored the action space sufficiently. The common support or overlap condition requires that for any context where the target policy assigns a non-zero probability to an action, the logging policy must also have a non-zero probability of selecting that action:

• Violation Consequence: If the target policy selects actions the logging policy never tried, the estimate becomes biased and unreliable
• Detection: Inspect the distribution of propensity scores and importance weights for extreme values
• Mitigation: Clip importance weights or use self-normalized estimators to trade some bias for reduced variance

Understanding the distinction between evaluation paradigms is critical for safe model deployment:

• On-Policy Evaluation: The model being evaluated is the same model that collected the data. Requires no correction but demands live deployment
• Off-Policy Evaluation: The target model differs from the data-collecting logging model. Enables safe, pre-deployment assessment but requires statistical correction
• A/B Testing: The gold-standard on-policy method that randomly assigns users to control and treatment groups, directly measuring causal impact without modeling assumptions
• Counterfactual Advantage: Allows rapid iteration on model candidates without the business risk or time cost of live experiments

High variance in off-policy estimates can render them useless for decision-making. Key techniques to stabilize estimates include:

• Weight Clipping: Capping importance weights at a maximum threshold to prevent any single sample from dominating the estimate
• Self-Normalized IPS: Dividing by the sum of importance weights rather than the sample size N, trading a small amount of bias for dramatic variance reduction
• Control Variates: Subtracting a baseline function from the reward that has a known expected value under the target policy, reducing variance without introducing bias
• Bootstrapped Confidence Intervals: Resampling the logged data to construct empirical confidence intervals around the point estimate

The simplest off-policy estimator that trains a regression model to predict the reward for any context-action pair, then evaluates the target policy by averaging the predicted rewards for the actions it would have selected. DM has low variance because it uses smooth model predictions, but it is biased if the reward model is misspecified. It ignores the actual observed rewards entirely during evaluation, relying solely on the learned model. Often used as a baseline or as the model component within a doubly robust estimator.

A variance reduction technique applied to IPS estimators. When the logging policy assigned a very small probability to an action, the inverse weight becomes extremely large, causing exploding variance. Clipping caps these weights at a maximum threshold (e.g., 20), trading a small amount of bias for a dramatic reduction in variance. This is a standard production safeguard. The self-normalized IPS variant divides by the sum of weights instead of the count of samples, providing implicit regularization.

The phenomenon where small per-step differences between the logging policy and the target policy compound over a trajectory, causing the state distribution visited by the target policy to diverge significantly from the logged data. In long-horizon settings like session-based recommendations, this makes standard off-policy estimators exponentially biased. Solutions include step-wise importance sampling and stationary distribution corrections that model the long-term visitation probabilities.

Statistical methods for quantifying the uncertainty of counterfactual estimates. Techniques include bootstrap resampling of logged trajectories, Bernstein-type inequalities that account for the heavy tails of importance-weighted estimators, and empirical Bernstein bounds that adapt to observed variance. Reporting a point estimate without a confidence interval is considered unreliable; production evaluation pipelines must output both the estimated value and its 95% confidence bounds to support safe deployment decisions.