Inverse Propensity Scoring (IPS)

IPS provides a statistically unbiased estimate of a target policy's value when the logging policy's action probabilities are known and non-zero for all actions the target policy might select. This property is critical for safe, offline model validation.

• Mechanism: Reweights each observed reward by 1 / π_logging(action | context)
• Requirement: Full knowledge of the logging policy's probability distribution
• Failure Mode: Bias is introduced if propensities are estimated rather than recorded exactly

IPS enables rigorous counterfactual evaluation of policies that exploit aggressively, even when the data was collected under a randomized or exploratory logging policy. This decouples data collection from evaluation.

• Exploration data can be reused to evaluate a purely exploitative target policy
• Variance increases as the target policy diverges from the logging policy
• Practical use: Evaluate a deterministic production policy on data collected during an A/B test

The primary weakness of IPS is high variance, especially when the logging policy assigns a very small probability to an action the target policy favors. The importance weight 1/π can become extremely large, dominating the estimate.

• Weight clipping: Capping importance weights at a threshold M introduces bias but dramatically reduces variance
• Self-normalization: Dividing by the sum of weights stabilizes estimates
• Effective sample size: Often much smaller than the actual dataset due to weight concentration

For IPS to be valid, the logging policy must have common support with the target policy. Every action the target policy might take must have a non-zero probability under the logging policy.

• Violation: If π_target(a|x) > 0 but π_logging(a|x) = 0, the estimator is undefined
• Practical implication: Logging policies must maintain some exploration over the full action space
• Mitigation: Use epsilon-greedy or Thompson Sampling during data collection to ensure full support

IPS is one of three core off-policy estimators, each with distinct bias-variance profiles. Understanding their trade-offs is essential for selecting the right evaluation strategy.

• Direct Method (DM): Learns a reward model r̂(a,x). Low variance but high bias if the model is misspecified.
• IPS: Zero bias with known propensities, but high variance.
• Doubly Robust (DR): Combines IPS and DM. Remains unbiased if either the propensity model or the reward model is correct, offering a practical safety net.

IPS is the standard tool for off-policy evaluation in contextual bandit systems, allowing data scientists to compare new model candidates against a production champion without risking live traffic.

• Champion-Challenger: Evaluate a challenger policy on historical data collected by the champion
• Model selection: Rank multiple candidate policies by their IPS-estimated value before shadow deployment
• Integration: Often paired with bootstrapping to construct confidence intervals around the IPS estimate