Off-Policy Evaluation

The foundational off-policy estimator that corrects for selection bias by re-weighting each observed outcome by the inverse of its probability of being selected by the logging policy.

• Mechanism: Reweights a logged event where action a was taken with probability π_b(a|x) by a factor of 1/π_b(a|x).
• Key Property: Provides an unbiased estimate of the target policy's value, assuming the logging policy has full support (i.e., it never assigns a zero probability to an action the target policy might take).
• Primary Challenge: Suffers from high variance, especially when the target and behavior policies differ significantly, leading to large, unstable weights that dominate the estimate.

A variance-reduction technique that caps the maximum importance weight to prevent any single logged event from dominating the estimator.

• Trade-off: Introduces a small amount of bias into the estimate in exchange for a dramatic reduction in variance, often resulting in a lower mean squared error.
• Implementation: Weights are capped at a threshold M, so the weight becomes min(π_t(a|x)/π_b(a|x), M).
• Practical Use: A standard first step in making IPS viable for production-scale datasets where propensity scores can be very small, leading to exploding weights.

A sophisticated estimator that combines a direct method reward model with IPS re-weighting to provide two layers of error correction.

• Core Formula: Combines the predicted reward from a regression model with the IPS-weighted residual error between the observed and predicted reward.
• Key Guarantee: The estimate is consistent (asymptotically unbiased) if either the propensity model or the reward model is correctly specified, providing a crucial safety net.
• Advantage: Often achieves the lowest variance among unbiased estimators when both models are reasonably well-specified, making it the gold standard for offline policy evaluation.

A biased but often lower-variance variant that normalizes the IPS estimate by the sum of the importance weights.

• Calculation: Divides the standard IPS estimate by the average importance weight, which forces the effective sample size to be accounted for.
• Bias-Variance Profile: While technically biased, the bias diminishes as sample size increases, and the variance reduction is often substantial in finite samples.
• Application: Particularly effective when importance weights have a large variance, as the normalization prevents the estimate from being skewed by a few extremely large weights.

The simplest off-policy approach that trains a supervised model to predict the reward for any context-action pair, then evaluates the target policy by querying this model.

• Process: A regression model r̂(x, a) is fit on the logged data. The target policy's value is estimated by averaging r̂(x, π_t(x)) over the logged contexts.
• Critical Flaw: Suffers from model misspecification bias. If the reward model is inaccurate, the estimate will be systematically wrong, with no statistical mechanism to correct for the error.
• Use Case: Often used as a baseline or as the reward model component within the more robust Doubly Robust estimator.

Essential metrics for validating the reliability of an off-policy estimate before trusting it for a deployment decision.

• Effective Sample Size (ESS): Estimates how many independent samples the weighted estimator is effectively based on. A low ESS relative to the actual sample size indicates the estimate is driven by a few high-weight events and is unreliable.
• Weight Distribution Analysis: Inspecting the empirical CDF of the importance weights. A heavy tail indicates high variance and potential instability.
• Policy Divergence Check: Quantifying the difference between the target and behavior policies (e.g., via KL-divergence) to preemptively identify scenarios where off-policy evaluation will be difficult.

The foundational off-policy estimator that corrects for selection bias in logged data. It re-weights observed rewards by the inverse probability of the logging policy selecting that action.

• Mechanism: If a logging policy chose action 'A' with probability 0.1, that event is weighted 10x to represent the general population.
• Variance Problem: Can suffer from high variance when logging propensities are small.
• Clipping: Propensity weights are often capped to stabilize estimates.

A hybrid estimator combining Inverse Propensity Scoring with a Direct Method reward model. It provides an unbiased estimate if either the propensity model or the reward model is correctly specified.

• Error Term: Corrects the direct method's prediction using the IPS-weighted residual.
• Safety Net: Offers a significant advantage over pure IPS when propensities are extreme.
• Use Case: Standard choice for high-stakes policy validation in e-commerce.

The broader statistical discipline of answering 'what if' questions using factual data. In bandits, it estimates how a target policy would have performed if it had been running instead of the behavior policy.

• Fundamental Challenge: We only observe the outcome of the taken action, never the alternatives.
• Assumption: Requires overlap (common support) between the logging and target policies.
• Application: Validating a new recommendation algorithm on last month's traffic logs.

A partial-information learning signal where only the reward for the chosen action is observed. The outcomes of all other actions remain unknown.

• Contrast: Differs from supervised learning where the correct label is always provided.
• Impact: This missing data is the root cause of the exploration-exploitation trade-off.
• Logging: Historical bandit feedback is the raw material for off-policy evaluation.

The optimization objective measuring the cumulative difference between the reward of the optimal oracle policy and the reward accumulated by the learning algorithm.

• Goal: Drive long-term regret as low as possible.
• Off-Policy Link: OPE estimates the expected regret of a new policy without incurring it live.
• Metric: Often visualized as a curve flattening over time as the algorithm learns.

A safe evaluation pattern where a challenger model runs in parallel with the production model on live traffic. It logs predictions and outcomes without affecting the user experience.

• Comparison: Shadow deployment collects unbiased data for future OPE, while OPE analyzes past data.
• Risk: Zero user-facing impact, making it ideal for high-risk verticals.
• Pipeline: Requires a dedicated logging infrastructure to capture the shadow model's decisions.