Minimum Detectable Effect

The relationship between MDE and sample size is governed by an inverse square law. Halving the MDE (e.g., from 2% to 1%) does not double the required sample size—it quadruples it.

• Practical Impact: Detecting a 0.5% lift requires 16x the traffic of a 2% lift.
• Business Constraint: This non-linear cost forces a hard trade-off between experimental sensitivity and traffic availability.
• Formula Context: The required sample size ( n ) is proportional to ( 1 / \text{MDE}^2 ).

MDE is often conflated with the significance level ((\alpha)), but they control distinct error sources. Statistical significance guards against false positives (Type I error), while MDE is calibrated against false negatives (Type II error).

• (\alpha) (Significance): Probability of detecting an effect when none exists. Typically set to 5%.
• (\beta) (Power): Probability of correctly detecting a true effect of size MDE. Typically set to 80% or 90%.
• Key Distinction: You can have a highly significant result (low p-value) for an effect smaller than the MDE, but the experiment was not powered to reliably detect it.

MDE is not an absolute number; it scales directly with the baseline variance of the target metric. Metrics with high natural volatility require a larger MDE for the same sample size.

• Low Variance Metrics: Click-through rate (CTR) on a stable, high-traffic widget allows for a very small MDE (e.g., 0.1%).
• High Variance Metrics: Average order value (AOV) or revenue per user, which have heavy-tailed distributions, demand a larger MDE (e.g., 2-5%) to avoid underpowered tests.
• Variance Reduction: Techniques like CUPED (Controlled-experiment Using Pre-Experiment Data) reduce variance, effectively lowering the achievable MDE without increasing traffic.

The MDE must be larger than the minimum business impact required to justify the engineering cost of shipping a model. An MDE set purely by statistical convenience risks detecting statistically significant but operationally irrelevant changes.

• Cost-Benefit Analysis: If a new personalization model requires 6 months of engineering effort, the MDE should reflect the revenue lift needed to generate a positive ROI within a fiscal year.
• Baseline Stability: The MDE must exceed the natural week-over-week fluctuation of the metric to avoid chasing noise.
• Guardrail Alignment: The MDE for a North Star Metric (e.g., revenue) should be considered alongside guardrail metrics (e.g., latency) to ensure the detected improvement doesn't mask a degradation elsewhere.

MDE directly determines the minimum runtime of an experiment. A smaller MDE requires capturing more user-week cycles to account for day-of-week effects and novelty effects.

• Novelty Effect: Users may initially engage with a new recommendation module regardless of its true quality. The experiment must run long enough for this effect to decay, which requires a sample size sufficient to detect the post-novelty MDE.
• Holdout Validation: Long-term holdout groups are essential for validating that the MDE observed in a 2-week experiment holds over a 6-month period without degrading due to user learning or ecosystem changes.

Traditional fixed-horizon MDE calculations assume a single evaluation at the end of the test. Sequential analysis and always-valid p-values allow for continuous monitoring, but they alter the effective MDE.

• Alpha Spending: Continuous monitoring requires adjusting significance thresholds to control the false positive rate, which slightly inflates the required sample size for a given MDE.
• Stopping Rules: If an experiment is stopped the moment the MDE is crossed, the effect size estimate is likely upwardly biased due to random variation. A post-selection inference correction is required.

The probability that an experiment will correctly reject a false null hypothesis. It is the complement of the Type II Error rate (β), calculated as 1 - β. An industry standard is 80% power, meaning there is a 20% chance of missing a true effect. Power is a direct function of the Minimum Detectable Effect, sample size, and variance. Increasing the MDE for a fixed sample size increases power, but risks missing smaller, practically significant improvements.

A pre-experiment calculation that determines the required sample size to detect a specified Minimum Detectable Effect with a given statistical power and significance level (α). Key inputs include:

• Desired Power (1-β): Typically 80%.
• Significance Level (α): Typically 5%.
• Baseline Conversion Rate: The current metric's average.
• MDE: The smallest lift you care about. Without this analysis, experiments often end prematurely, leading to underpowered, inconclusive results.

A standardized, scale-free measure of the magnitude of a phenomenon. Unlike the raw Minimum Detectable Effect, which is often expressed in absolute percentage points (e.g., a 2% lift), Cohen's d or Glass's Δ provide a universal measure of practical significance. This allows for comparison across different metrics. A small MDE in raw terms might correspond to a trivial effect size, questioning the business value of detecting it.

A false negative error where the experiment fails to detect a true effect, retaining the null hypothesis incorrectly. The probability of this error is denoted by β. The Minimum Detectable Effect is intrinsically linked to β; setting a very small MDE without a proportional increase in sample size will inflate β, making it highly likely you will miss the very effect you are trying to detect. Balancing MDE and β is the core trade-off in experimental design.

A critical validity check, not a statistical parameter, but a failure that invalidates MDE calculations. SRM occurs when the observed traffic split deviates significantly from the intended randomization ratio (e.g., 50/50). If the sample size is corrupted by a bug in the randomization pipeline, the pre-calculated Minimum Detectable Effect is no longer valid, and the test's power is compromised. It is the first guardrail metric checked before analyzing results.

A conservative adjustment to the significance level (α) when testing multiple metrics or variants. If you test 20 metrics, the probability of a false positive increases. The Bonferroni correction divides α by the number of tests (e.g., 0.05 / 20 = 0.0025). This directly impacts the Minimum Detectable Effect, as a stricter α requires a much larger sample size to maintain the same power, effectively increasing the MDE for a fixed traffic volume.