Power Analysis

The probability that an experiment will correctly reject the null hypothesis when a true effect exists. It represents the test's sensitivity to detect real differences.

• Standard target: 0.80 (80%) is the conventional minimum in most experimentation platforms
• Interpretation: An 80% power means there is a 20% chance of a Type II Error (false negative)
• Trade-off: Higher power requires larger sample sizes, increasing experiment duration and cost
• Business impact: Underpowered tests waste engineering resources by failing to detect genuine improvements in personalization models

Setting power too low risks missing revenue-positive model updates; setting it too high may make experimentation infeasible for low-traffic segments.

The probability threshold for committing a Type I Error—rejecting the null hypothesis when it is actually true. This is the experimenter's tolerance for false positives.

• Standard value: 0.05 (5%) is the conventional cutoff in online controlled experiments
• One-tailed vs. two-tailed: A two-tailed test allocates α/2 to each tail, requiring more data but detecting effects in either direction
• Relationship to p-value: If the calculated p-value falls below α, the result is declared statistically significant
• Multiple testing: When monitoring multiple metrics, α must be adjusted using corrections like Bonferroni or False Discovery Rate control

A lower α reduces false positives but demands larger sample sizes, directly impacting the feasibility of rapid iteration on AI models.

The smallest effect size that the experiment is designed to reliably detect. This is the practical significance threshold—the improvement that justifies shipping a change.

• Expression: Often stated as a relative percentage change (e.g., 2% lift in conversion rate)
• Business calibration: MDE should reflect the smallest improvement that delivers meaningful return on investment
• Inverse relationship: Smaller MDEs require exponentially larger sample sizes
• Example: Detecting a 0.5% click-through rate improvement requires roughly 4x the sample size of detecting a 1% improvement

Setting the MDE requires collaboration between data scientists and product managers to align statistical requirements with business objectives. An overly ambitious MDE can make experiments practically impossible to run.

The mathematical derivation of the number of experimental units required per variant, computed from power, significance level, MDE, and the baseline variance of the target metric.

• Key inputs: α (significance), 1-β (power), MDE, and the standard deviation of the metric
• Binary metrics: For conversion rates, variance is a function of the baseline rate itself: σ² = p(1-p)
• Continuous metrics: For revenue or latency, historical variance must be estimated from production data
• Duration estimation: Sample size divided by daily eligible traffic determines experiment runtime

Formula for two-sample proportion test:

codeCopy
n = (Z_α/2 + Z_β)² × (p₁(1-p₁) + p₂(1-p₂)) / (p₂ - p₁)²

Online calculators and libraries like statsmodels in Python automate this computation, but understanding the inputs ensures correct parameterization.

The current performance level of the control group metric, which directly influences the required sample size through its effect on variance.

• Variance peak: Binary metrics exhibit maximum variance at a 50% baseline rate, requiring the largest samples
• Low baselines: Rare events (e.g., 0.1% purchase rate) require massive sample sizes because the absolute effect is tiny
• Stability requirement: Baseline estimates must come from a stable historical period, not a volatile launch window
• Seasonality adjustment: Baselines should account for day-of-week and holiday effects to avoid misestimating variance

Accurate baseline measurement is critical—an underestimated variance leads to underpowered experiments that fail to detect real effects from personalization model improvements.

A visualization plotting the relationship between sample size and statistical power for a fixed MDE and significance level. It reveals the diminishing returns of additional data collection.

• Shape: The curve rises steeply initially, then asymptotically approaches 1.0
• Saturation point: Beyond 90-95% power, massive sample size increases yield negligible gains
• Practical use: Helps stakeholders understand the cost of increasing power from 80% to 90% or 95%
• Inverse analysis: Can also plot detectable effect size against sample size at fixed power to determine what effect sizes are discoverable given traffic constraints

Power curves are essential communication tools for aligning engineering, product, and data science teams on realistic experiment expectations and timelines.

The two fundamental error classes in hypothesis testing. Type I Error (False Positive): Concluding a variant works when it doesn't—controlled by the significance level α (typically 0.05). Type II Error (False Negative): Missing a real improvement—controlled by statistical power (1-β). The trade-off:

• Decreasing α reduces false positives but increases false negatives
• Power analysis balances these risks against sample size constraints
• In revenue-critical personalization, the cost of each error type must be explicitly modeled

A standardized, scale-free measure of the magnitude of difference between groups, independent of sample size. Unlike p-values, effect size answers 'how much' rather than 'whether.' Common measures:

• Cohen's d: For differences between two means
• Relative lift: Percentage improvement over control
• Absolute difference: In native metric units (e.g., +$0.03 ARPU) Always report effect size alongside significance—a statistically significant result with negligible effect size is practically meaningless.