Statistical Power

The magnitude of the difference between the control and treatment groups that you consider practically significant. A larger minimum detectable effect (MDE) requires less data to achieve the same power.

• A 5% lift in conversion is easier to detect than a 0.5% lift
• Effect size is standardized using metrics like Cohen's d to enable cross-experiment comparisons
• In retail personalization, small effect sizes (1-2%) are common and require massive sample sizes
• Setting an unrealistically small MDE leads to prohibitively long experiments that never conclude

The probability of committing a Type I error — rejecting the null hypothesis when it is actually true. This is the threshold for declaring statistical significance.

• Conventionally set at α = 0.05, meaning a 5% false positive risk
• Lowering alpha to 0.01 increases the required sample size but reduces false discoveries
• In large-scale experimentation platforms, stricter alpha values are used to control the False Discovery Rate
• The choice of alpha directly trades off between sensitivity and the cost of shipping a neutral or harmful variant

The number of experimental units — typically users, sessions, or transactions — allocated to each variant. Power increases monotonically with sample size.

• Doubling the sample size does not double power; the relationship follows a non-centrality parameter curve
• Required sample size is calculated via power analysis before the experiment launches
• In online retail, high-traffic pages may reach significance in hours; low-traffic segments may require weeks
• Insufficient sample size is the most common cause of underpowered experiments and Type II errors

The inherent noise or variability in the outcome metric being measured. High-variance metrics reduce statistical power by obscuring the treatment signal.

• Click-through rate has lower variance than average order value, making it easier to power experiments on CTR
• Variance reduction techniques like CUPED (Controlled-experiment Using Pre-Experiment Data) use pre-period covariates to adjust metrics and increase sensitivity
• Stratified sampling on key segments reduces variance by ensuring balanced treatment-control splits
• Metrics with heavy-tailed distributions require larger samples or winsorization to stabilize variance estimates

A pre-experiment calculation that determines the required sample size to detect a specified effect with a given probability. It balances four interconnected parameters: statistical power (typically 80%), significance level (α, typically 5%), effect size, and sample size. Fixing any three determines the fourth.

• Prevents underpowered experiments that waste resources
• Uses tools like G*Power or statsmodels in Python
• Critical input: the minimum detectable effect (MDE) you cannot afford to miss

A false negative error where the null hypothesis is incorrectly retained despite a true effect existing. Statistical power is defined as 1 - β, making it the direct complement of the Type II error rate.

• Occurs when sample sizes are too small or variance is too high
• Standard acceptable β is 0.20 (20% chance of missing a real effect)
• More dangerous in practice than Type I errors because improvements are silently abandoned

The smallest practically meaningful difference between control and treatment that an experiment is designed to reliably identify. Setting the MDE too small requires enormous sample sizes; setting it too large risks missing incremental but valuable improvements.

• Expressed in absolute terms (e.g., 2% lift in conversion) or standardized units (Cohen's d)
• Directly drives sample size calculations in power analysis
• Should be calibrated against business impact, not just statistical convenience

A standardized, scale-free measure of the magnitude of a phenomenon. Unlike p-values, effect size quantifies practical significance independent of sample size. Common measures include Cohen's d for mean differences and Cohen's h for proportions.

• Small: d = 0.2 | Medium: d = 0.5 | Large: d = 0.8
• Essential for comparing results across experiments with different metrics
• Large samples can produce significant p-values for trivially small effect sizes

The process of calculating how many experimental units (users, sessions, transactions) are required to achieve a target statistical power. The formula depends on the chosen α level, desired power (1-β), expected variance in the metric, and the minimum detectable effect.

• Larger sample sizes increase power by reducing standard error
• Diminishing returns: doubling sample size does not double power
• Online calculators and libraries like statsmodels.stats.power automate this computation

A range of values that likely contains the true population parameter with a specified confidence level (typically 95%). Wider intervals indicate lower precision and are a direct consequence of low statistical power or high variance.

• A 95% CI means: if the experiment were repeated infinitely, 95% of intervals would contain the true effect
• Narrow CIs signal high precision and sufficient power
• If the CI for a treatment effect excludes zero, the result is statistically significant at the corresponding α level