Statistical Significance

The p-value is the probability of observing results at least as extreme as the current experiment's results, assuming the null hypothesis (that there is no real effect) is true. It quantifies the evidence against the null hypothesis.

• A low p-value (typically ≤ 0.05) suggests the observed effect is unlikely under the null hypothesis, leading to its rejection.
• It is not the probability that the null hypothesis is true, nor the probability that the alternative hypothesis is false.
• In A/B testing, a p-value of 0.03 for a click-through rate improvement means there's a 3% chance of seeing such a difference if the new model (variant B) was actually no better than the old one (variant A).

The significance level, denoted by α (alpha), is the pre-defined probability threshold used to determine statistical significance. It represents the maximum risk of a Type I error (false positive) you are willing to accept.

• Common values are 0.05 (5%) or 0.01 (1%).
• If the calculated p-value is less than or equal to α, the result is deemed statistically significant.
• Setting α = 0.05 implies a 5% chance of incorrectly declaring a winner when no true difference exists. This threshold must be set before the experiment begins to avoid p-hacking.

Statistical power is the probability that a test will correctly reject a false null hypothesis. It measures the test's sensitivity to detect a true effect of a specified size.

• Power is calculated as 1 - β, where β is the probability of a Type II error (false negative).
• High power (typically ≥ 0.80 or 80%) is crucial for reliable experiments. Low power increases the risk of missing a real improvement.
• Power increases with larger sample sizes, larger true effect sizes, and higher significance levels (α).
• Before an A/B test, a power analysis is conducted to determine the required sample size to detect the Minimum Detectable Effect.

A confidence interval provides a range of plausible values for the true effect size (e.g., the difference in conversion rates), with a specified level of confidence (e.g., 95%). It offers more information than a binary significant/not-significant result.

• A 95% confidence interval means that if the experiment were repeated many times, 95% of the calculated intervals would contain the true population parameter.
• If a 95% CI for a lift in revenue per user is [$1.50, $3.00], we are 95% confident the true value lies within that range.
• Intervals that do not include zero (no effect) align with statistically significant p-values. Narrow intervals indicate more precise estimates.

Effect size is a quantitative measure of the magnitude of the observed difference or relationship. While statistical significance tells you if an effect exists, effect size tells you how large it is.

• Common measures include Cohen's d (standardized mean difference), relative difference (e.g., 10% lift), and absolute difference.
• A result can be statistically significant (due to a large sample) but have a trivial effect size with no practical business impact.
• In model evaluation, the Average Treatment Effect is a key effect size metric comparing the performance of treatment (new model) versus control.

The sample size (number of observations or users in an experiment) and the underlying variability (standard deviation) of the metric are fundamental determinants of statistical significance.

• Larger sample sizes reduce standard error, leading to more precise estimates, narrower confidence intervals, and higher statistical power.
• High variability in the outcome metric (e.g., highly variable user session times) makes it harder to detect a signal, requiring larger samples.
• Sample size calculation requires specifying the significance level (α), desired power (1-β), expected variability, and the minimum detectable effect considered meaningful.

The most direct application is in online A/B tests for model launches. For instance, when deploying a new large language model for a customer service chatbot, engineers randomly split user traffic. They compare the primary metric (e.g., user satisfaction score or issue resolution rate) between the new model (treatment) and the old model (control). A statistically significant result (p-value < 0.05) provides confidence that the observed improvement is real, not a random fluctuation, justifying a full rollout.

In model development, statistical tests determine if a new input feature genuinely improves predictive performance. After adding a feature, data scientists compare model accuracy on a hold-out test set. Using a paired t-test on the prediction errors of the old and new model, they assess if the accuracy gain is significant. This prevents overfitting and ensures engineering effort is spent on features that provide a measurable, reproducible lift.

Significance testing is core to drift detection systems. MLOps pipelines continuously monitor the statistical properties of live inference data versus the training data distribution. Tools use tests like the Kolmogorov-Smirnov test (for continuous features) or Chi-squared test (for categorical features). A statistically significant divergence triggers an alert, indicating the model's performance may be degrading and retraining or investigation is required.

When evaluating new model architectures (e.g., comparing a transformer to an LSTM), researchers run multiple training runs with different random seeds to account for variance. They then report average performance metrics with confidence intervals. If the 95% confidence intervals of two models do not overlap, it provides strong evidence of a statistically significant performance difference, guiding architectural decisions beyond single-run leaderboard scores.

In ethical AI audits, statistical significance tests identify disparate impact. Performance metrics (e.g., false positive rate) are calculated separately for different demographic groups. A statistical parity test determines if observed performance gaps are significant. For example, a significant difference in loan approval rates between groups, after controlling for relevant features, signals potential unfair bias that must be addressed before deployment.

While A/B tests use fixed traffic splits, adaptive experiments like multi-armed bandits use significance calculations to dynamically allocate traffic. Algorithms like Thompson Sampling maintain Bayesian posterior distributions for each variant's performance. They continuously evaluate the probability that one variant is significantly better, shifting traffic toward the best-performing option to maximize reward while still gathering enough data for confident decision-making.

A p-value is the probability, under the assumption of the null hypothesis (no effect), of obtaining a test statistic result at least as extreme as the one actually observed. It is the primary metric compared against the significance level (alpha) to determine statistical significance.

• A low p-value (typically < 0.05) provides evidence against the null hypothesis.
• It is not the probability that the null hypothesis is true, nor the probability that the alternative hypothesis is false.
• In A/B testing, it quantifies how surprising the observed difference between variants is, assuming there is no real difference.

Statistical power is the probability that a hypothesis test will correctly reject a false null hypothesis. It represents the test's sensitivity to detect a true effect if one exists.

• Power is calculated as 1 - β, where β is the probability of a Type II error (false negative).
• It is influenced by three key factors: sample size, effect size, and significance level (alpha).
• In practice, experiments are designed with a target power (often 80% or 90%) to ensure a reasonable chance of detecting a meaningful business impact.

A confidence interval is a range of values, derived from sample data, that is likely to contain the true value of an unknown population parameter (e.g., the true difference in conversion rates).

• A 95% confidence interval means that if the experiment were repeated many times, 95% of the calculated intervals would contain the true parameter.
• It provides more information than a binary significant/not-significant result by indicating the precision and potential magnitude of the observed effect.
• Narrow intervals indicate higher precision, often achieved with larger sample sizes.

The Minimum Detectable Effect is the smallest true effect size that an experiment is statistically powered to detect, given a specified sample size, significance level, and desired power.

• It is a critical input for sample size calculation before launching an A/B test.
• Choosing an MDE involves a business trade-off: a smaller MDE requires a larger sample size but can detect subtler improvements.
• For example, an experiment powered to detect an MDE of 2% in conversion rate needs more users than one powered to detect a 5% change.

These are the two fundamental categories of incorrect conclusions in statistical hypothesis testing.

• Type I Error (False Positive): Incorrectly rejecting a true null hypothesis. Concluding a difference exists when it does not. The probability of a Type I error is denoted by alpha (α), the significance level (e.g., 0.05).
• Type II Error (False Negative): Failing to reject a false null hypothesis. Missing a real difference. The probability of a Type II error is denoted by beta (β).
• The peeking problem in A/B testing inflates the Type I error rate by repeatedly checking results before a sample size is complete.

The null hypothesis is a default statistical proposition that there is no effect or no difference between groups. It is the assumption that an A/B test aims to challenge with evidence from sample data.

• In a standard A/B test, the null hypothesis states that the performance metric (e.g., conversion rate) is identical for the control (A) and treatment (B) variants.
• The goal of the test is to gather sufficient evidence to reject the null hypothesis in favor of the alternative hypothesis (that a difference exists).
• Statistical significance is formally the rejection of the null hypothesis based on the p-value.