Off-Policy Evaluation (OPE)

An estimator is unbiased if its expected value equals the true value of the target policy's performance. In OPE, unbiasedness is the gold standard, ensuring that estimation errors are not systematic. The Inverse Propensity Scoring (IPS) estimator is a classic example of an unbiased estimator, provided the logging policy is known and has full support. Achieving unbiasedness often comes at the cost of high variance, leading to the development of biased-but-stable alternatives.

Variance measures the variability of the estimator across different finite samples of historical data. A high-variance estimator is unreliable in practice, even if it is theoretically unbiased. The primary culprit for high variance in OPE is a large importance weight, which occurs when the target policy selects an action that the behavior policy rarely chose. Techniques like weight clipping and self-normalization are used to trade a small amount of bias for a dramatic reduction in variance.

This is the central tension in OPE. Purely unbiased methods like standard IPS often have unacceptable variance for real-world datasets. Conversely, methods that introduce bias, such as Doubly Robust (DR) estimation with a poorly fit outcome model, can be misleading. The goal is to find an estimator on the efficient frontier of this tradeoff. The Doubly Robust estimator is specifically designed to navigate this, remaining unbiased if either the propensity model or the outcome model is correctly specified.

The support of the behavior policy is the set of all actions it has a non-zero probability of taking. A fundamental requirement for most OPE methods is the common support or coverage assumption: the target policy can only choose actions that the behavior policy was capable of choosing. If the target policy selects an action that was never taken in the historical data, the importance weight is undefined, and evaluation is impossible without a reliable outcome model to extrapolate.

An estimator is consistent if it converges to the true performance value as the size of the historical dataset approaches infinity. This is a minimum requirement for any useful estimator. While an unbiased estimator is always consistent, a biased estimator can also be consistent if the bias diminishes as more data is collected. For example, a clipped IPS estimator is biased but consistent, as the clipping threshold can be relaxed with larger datasets.

Efficiency relates to the precision of an estimator. An efficient estimator achieves the lowest possible variance among a class of estimators, known as the Cramér-Rao lower bound. In OPE, the Doubly Robust estimator is often more efficient than IPS because it leverages an outcome model to reduce reliance on importance weights alone. A more efficient estimator provides tighter confidence intervals, enabling more decisive conclusions about whether a new policy is truly superior.

A foundational off-policy evaluation method that corrects for selection bias in logged data. It re-weights observed outcomes by the inverse probability of the action being taken by the logging policy.

• Mechanism: If a logging policy chose action 'A' with probability 0.1, that event's weight is 10x in the new policy's estimate.
• Key Requirement: The logging policy must have a non-zero probability of taking any action the target policy might take (common support).
• Variance Problem: IPS estimators can suffer from high variance when propensity scores are small, leading to unstable estimates.

A powerful hybrid method that combines Inverse Propensity Scoring with a direct outcome model to provide an unbiased estimate of a policy's value.

• Consistency Guarantee: Remains consistent if either the propensity model or the outcome model is correctly specified, providing a safety net.
• Mechanism: Uses the outcome model as a baseline and applies IPS to correct the residuals, reducing variance compared to pure IPS.
• Application: The gold standard for off-policy evaluation in high-stakes domains like healthcare and finance where model misspecification is a real risk.

The simplest off-policy evaluation approach that trains a reward predictor on historical data and uses it to estimate the value of the target policy.

• Mechanism: For each context in the log, predict the reward for the action the target policy would take, then average.
• Critical Flaw: Suffers from high bias if the reward model is misspecified, as it does not correct for the distributional shift between the logging and target policies.
• Use Case: Often used as a baseline or as the outcome model component within a Doubly Robust estimator.

Techniques to assess the reliability of off-policy estimates by analyzing the importance weights themselves.

• Effective Sample Size (ESS): Estimates how many independent samples the weighted data is equivalent to. A low ESS warns that the estimate is driven by a few high-weight examples.
• Weight Clipping: A variance reduction technique that caps extreme importance weights at a maximum threshold, introducing some bias to dramatically reduce variance.
• Per-Decision Importance Sampling: A variant that multiplies importance weights at each step of a trajectory, critical for evaluating policies in sequential decision-making problems.

Off-policy evaluation is fundamentally a causal inference problem. The logging policy introduces selection bias—the actions taken are not independent of the context.

• Confounding: The context (e.g., user history) influences both the logging policy's action and the outcome. OPE methods must de-confound this relationship.
• Ignorability Assumption: Assumes all variables that affect both action selection and the outcome are observed in the logged data.
• A/B Testing Comparison: Unlike randomized A/B tests, OPE must statistically correct for the non-random assignment of actions in observational data.