Statistical Power

The effect size quantifies the magnitude of the difference or relationship you aim to detect. It is the standardized measure of the true signal in your data.

• Larger effects are easier to detect, requiring smaller sample sizes to achieve the same power.
• Smaller effects require significantly larger samples. For example, detecting a 1% lift in a conversion rate requires far more users than detecting a 10% lift.
• Common measures include Cohen's d (for mean differences) and odds ratios (for proportions).

In practice, the minimum detectable effect is a critical planning parameter, representing the smallest effect size your experiment is powered to find.

The sample size is the number of independent observations or experimental units (e.g., users, sessions) in your study.

• Power increases with sample size. More data reduces the standard error of your estimate, making it easier to distinguish a true effect from random noise.
• The relationship is non-linear; doubling power often requires more than doubling the sample size.
• In online A/B testing, sample size is directly tied to traffic volume and experiment duration. Underpowered tests (too few samples) are a primary cause of inconclusive or misleading results.

Statistical power calculations are fundamentally used to determine the necessary sample size before an experiment begins.

The significance level, denoted by alpha (α), is the probability threshold for rejecting the null hypothesis when it is actually true—a Type I error or false positive.

• It is a pre-defined risk tolerance for false alarms, commonly set at 0.05 (5%).
• A lower alpha (e.g., 0.01) reduces false positives but also reduces statistical power for a given sample size and effect size, as the evidence required to reject the null becomes stricter.
• The choice of alpha involves a trade-off between sensitivity and risk, often guided by the cost of a false positive decision in the specific business context.

The choice of statistical test (e.g., t-test, chi-squared test) and the underlying variability in your data directly constrain power.

• Tests must be appropriate for your data type (continuous, binary) and experimental design.
• High variance (noise) in the outcome metric obscures the signal, reducing power. Techniques like stratified sampling or CUPED can reduce variance to increase sensitivity.
• One-tailed vs. two-tailed tests: A one-tailed test (directional hypothesis) has more power to detect an effect in a specified direction than a two-tailed test, but cannot detect effects in the opposite direction.

Optimizing the test and minimizing unnecessary variability are key engineering levers for efficient experimentation.

Statistical power is the primary driver for calculating the required sample size in an A/B test comparing two AI models. Before launching an experiment, practitioners must specify:

• Desired power (typically 80% or 90%)
• Significance level (alpha) (typically 5%)
• Minimum Detectable Effect (MDE) (the smallest performance lift considered meaningful)

The required sample size increases with higher desired power, a lower alpha, and a smaller MDE. Underpowered tests (low sample size) are a major pitfall, as they lack the sensitivity to detect real improvements, leading to false negatives and wasted R&D effort.

When an A/B test fails to reject the null hypothesis (shows no statistically significant difference), statistical power provides critical context. An inconclusive result can mean:

• There is truly no effect (the models perform identically).
• The test was underpowered and failed to detect a real effect.

Post-hoc power analysis can estimate the probability the test had to detect the observed effect size. If this probability is low (e.g., < 30%), the result is unreliable, and the experiment may need to be re-run with a larger sample. This prevents incorrectly shelving a superior model.

When evaluating a new model against a benchmark suite, power analysis ensures performance comparisons are meaningful. For example, if Model A scores 92.1% accuracy and Model B scores 92.3% on a test set of 10,000 examples, is the 0.2% difference real or noise?

A power analysis calculates the effect size of this difference relative to the variance. If the test's power to detect this small effect is low, the benchmark suite may be insufficiently large to declare a winner. This informs decisions to collect more evaluation data or to use more sensitive statistical tests.

In live A/B tests, the primary goal is often to maximize a key performance indicator (KPI) like click-through rate. However, guardrail metrics (e.g., latency, fairness scores, user retention) must also be monitored. Statistical power considerations apply here as well.

A test powered to detect a 1% lift in the primary KPI may be severely underpowered to detect a 0.5% degradation in a critical guardrail metric. Teams must conduct multivariate power analyses or use sequential testing frameworks that monitor multiple endpoints, ensuring the experiment is sensitive enough to detect harmful side effects before full rollout.

Sequential testing methods allow for evaluating experiment results as data accumulates, enabling early stopping. This interacts directly with statistical power.

• The Peeking Problem: Repeatedly checking p-values inflates Type I error (false positives).
• Sequential Designs: Methods like Alpha-spending functions (e.g., O'Brien-Fleming) control the overall error rate, allowing for interim analyses while preserving power.

These designs are crucial for AI testing, where models can be deployed rapidly. They allow teams to stop a test early if a new model is clearly superior or clearly harmful, optimizing resource use while maintaining statistical rigor.

Statistical power is essential for drift detection systems that monitor model performance in production. These systems run continuous hypothesis tests comparing recent performance against a baseline.

• Null Hypothesis: No performance drift has occurred.
• Alternative Hypothesis: Performance has degraded.

The detection sensitivity of these systems is their statistical power. Configuring them requires specifying the effect size of meaningful drift (e.g., a 2% drop in accuracy) and the desired power to detect it. An underpowered monitor will fail to alert on significant degradation, leading to prolonged service quality issues.

A determination that an observed difference between experimental variants is unlikely to be due to random chance. It is formally assessed by comparing a calculated p-value to a pre-defined significance level (alpha), commonly set at 0.05. Achieving statistical significance is the primary goal of a hypothesis test, but it must be interpreted alongside effect size and confidence intervals to understand practical importance.

The probability, assuming the null hypothesis is true, of observing a test result at least as extreme as the one obtained from the experiment. A small p-value (e.g., < 0.05) provides evidence against the null hypothesis.

• Misinterpretation Risk: A p-value is not the probability the null hypothesis is true, nor the probability the alternative hypothesis is false.
• Context is Critical: A p-value must be considered with sample size and minimum detectable effect; a tiny effect can be 'significant' with a huge sample.

The smallest true effect size that an experiment is designed to detect with a specified level of statistical power (e.g., 80%). It is a critical input for sample size calculation.

• Trade-off: A smaller MDE requires a larger sample size to maintain power.
• Practical Setting: The MDE should be set based on business impact, not statistical convenience. Detecting a 0.1% lift in conversion may require an impractically large experiment.

A range of values, calculated from sample data, that is likely to contain the true value of an unknown population parameter (e.g., the true difference in means between variants). A 95% confidence interval means that if the experiment were repeated many times, 95% of such intervals would contain the true parameter.

• More Informative than P-Value: Provides an estimate of the effect size and its precision.
• Direct Relationship to Power: A higher-power experiment will typically yield a narrower confidence interval.

The two fundamental errors in hypothesis testing.

• Type I Error (False Positive): Incorrectly rejecting a true null hypothesis. The probability of this is controlled by the significance level (alpha).
• Type II Error (False Negative): Failing to reject a false null hypothesis. The probability of this is denoted by beta.

Statistical power is defined as 1 - beta, the probability of correctly rejecting a false null and avoiding a Type II error.

The process of determining the number of observations or experimental units needed to achieve a desired statistical power to detect a specified minimum detectable effect (MDE) at a given significance level. The formula incorporates:

• Significance Level (Alpha): Typically 0.05.
• Power (1 - Beta): Typically 0.8 or 0.9.
• Baseline Metric Rate: For proportion metrics like conversion rate.
• Variance/Standard Deviation: For continuous metrics.

Underpowered experiments (too small sample) are a primary cause of inconclusive or misleading A/B test results.