Minimum Detectable Effect

Statistical power is the probability that your test will correctly detect a true effect of the specified MDE size. It represents the test's sensitivity.

• Standard Threshold: 80% or 90%. A power of 80% means there's a 20% chance of a Type II error (false negative).
• Trade-off with Sample Size: Higher power requires a larger sample size, all else being equal. Doubling power from 80% to 90% significantly increases the required N.
• Engineering Implication: Choosing 80% vs. 90% power is a business risk decision balancing the cost of a missed opportunity against the cost of running a larger, longer experiment.

The significance level (alpha) is the probability threshold for rejecting the null hypothesis when it is actually true, i.e., the risk of a Type I error (false positive).

• Standard Threshold: 5% (α = 0.05). This sets a 5% false positive rate.
• Relationship to MDE: A stricter alpha (e.g., 0.01) requires stronger evidence to declare a winner, which increases the required sample size for a given MDE and power.
• Multiple Testing Correction: If running many concurrent experiments or checking multiple metrics, the family-wise error rate inflates. Techniques like the Bonferroni correction adjust alpha downward, which directly increases the MDE or required sample size.

The baseline conversion rate is the current performance metric of your control variant before any change. For a binary metric (e.g., click-through rate), this is a proportion (p).

• Critical Input: MDE is often expressed as a relative lift (e.g., 5%). The absolute difference the test must detect is p * (MDE relative). A 5% lift on a 10% baseline is a 0.5 percentage point absolute change.
• Impact on Variance: The variance of a proportion is p(1-p). This variance is highest when p = 0.5 and decreases as p approaches 0 or 1. Higher variance requires a larger sample size to detect the same relative MDE.
• Practical Note: Use recent, reliable historical data to estimate p. An inaccurate baseline is a common cause of underpowered experiments.

Sample size (N) is the total number of independent experimental units (e.g., users, sessions) required. It is the output of the MDE calculation but also a primary constraint.

• Calculation: N is derived from the chosen α, power (1-β), baseline rate (p), and the absolute MDE. Formulas differ for proportions (z-test) and means (t-test).
• 50/50 Split: The most statistically efficient allocation is an equal split between control and treatment. Deviating from this (e.g., 90/10) increases the total N required for the same power.
• Traffic & Duration: N must be feasible given your available traffic. Required experiment duration = N / (daily traffic * allocation %). Long durations increase the risk of seasonal effects contaminating results.

For continuous metrics (e.g., revenue per user, session duration), the variance (σ²) or standard deviation (σ) of the underlying data is a key driver of MDE.

• Role in Calculation: The detectable difference in means is proportional to σ / √N. Noisier data (high σ) makes it harder to detect small effects, requiring a larger N or accepting a larger MDE.
• Estimation Challenge: Accurately estimating σ from historical data is crucial. Overestimating σ leads to an overpowered, wasteful test; underestimating leads to an underpowered test likely to miss real effects.
• Reducing Variance: Techniques like CUPED (Controlled-experiment Using Pre-Experiment Data) can reduce σ by accounting for user-level covariates, effectively lowering the MDE for a fixed N.

This choice defines the alternative hypothesis. A two-sided test looks for any difference (increase or decrease). A one-sided test looks for a difference in a specific direction (e.g., increase only).

• Statistical Impact: A one-sided test at alpha = 0.05 has the same critical value as a two-sided test at alpha = 0.10. Therefore, for the same α, a one-sided test has higher power (or a smaller MDE) to detect an effect in the specified direction.
• Appropriate Use: One-sided tests are only valid when you have a strong a priori reason that the effect cannot be in the opposite direction (e.g., removing a latency bug cannot increase latency). Misuse increases the risk of false positives.
• Standard Practice: Two-sided tests are the default in A/B testing for fairness, as they guard against unexpected negative impacts.

Statistical power is the probability that a hypothesis test will correctly reject a false null hypothesis. It represents the test's sensitivity to detect a true effect when one exists.

• Directly linked to MDE: A higher desired power (e.g., 80% vs. 90%) requires a larger sample size to detect the same MDE, or allows detection of a smaller MDE with the same sample size.
• Calculated as 1 - β, where β is the probability of a Type II error (failing to detect a true effect).
• Along with significance level (α) and sample size, power is a key input for calculating the MDE before an experiment begins.

Sample size is the number of observations or experimental units (e.g., users, sessions) included in a study. It is a primary determinant of an experiment's precision and its ability to detect an effect.

• Has an inverse relationship with MDE: For a fixed power and significance level, a larger sample size enables the detection of a smaller Minimum Detectable Effect.
• Sample size calculations are performed during experiment planning to ensure the test is adequately powered to detect the business-relevant MDE.
• Insufficient sample size leads to underpowered experiments that are likely to miss meaningful effects, resulting in inconclusive or misleading 'no significant difference' findings.

Effect size is a quantitative measure of the magnitude of a phenomenon or the difference between two groups. Unlike statistical significance, it is not influenced by sample size.

• The MDE is a planned or target effect size. It is the threshold of practical importance that the experiment is designed to detect.
• Common measures include Cohen's d (standardized mean difference), relative risk, and percentage lift (e.g., a 2% increase in conversion rate).
• After an experiment, the observed effect size is calculated from the data. If the observed effect meets or exceeds the pre-defined MDE and is statistically significant, the result is considered both reliable and meaningful.

The significance level, denoted by alpha (α), is the probability of rejecting the null hypothesis when it is actually true (a Type I error or false positive).

• It is the threshold for declaring statistical significance. A common standard is α = 0.05, meaning a 5% risk of a false positive.
• Along with power and sample size, α is a critical input for calculating the MDE. A more stringent alpha (e.g., 0.01) requires a larger sample size to detect the same MDE, or results in a larger MDE for a fixed sample size.
• Setting α involves a trade-off between the risk of false positives and the sensitivity (power) of the test.

Practical significance asks whether a statistically detected effect is large enough to be of real-world value or worth the cost of implementation. It is a business judgment, not a statistical one.

• The MDE is explicitly defined to represent practical significance. An experiment should be powered to detect the smallest effect that would justify a business decision (e.g., launching a new model).
• A result can be statistically significant (unlikely due to chance) but not practically significant if the observed effect is tiny and below the MDE threshold.
• Defining the MDE forces alignment between data scientists and product/business stakeholders on what constitutes a meaningful outcome before the experiment runs.

A Type II error occurs when a hypothesis test fails to reject a false null hypothesis, meaning a true effect is missed. The probability of a Type II error is denoted by beta (β).

• Statistical power is 1 - β. If β=0.20, power is 0.80 (80%).
• The MDE is defined in relation to β. The test is designed to have a (1-β) probability of detecting an effect as large as or larger than the MDE.
• A high β (low power) increases the risk of concluding 'no difference' when a meaningful difference (at or above the MDE) actually exists. This is a primary risk of underpowered experiments.