Statistical Significance

The null hypothesis is the default assumption that there is no effect or no difference between groups. In A/B testing, it posits that the new model (variant B) performs identically to the baseline (variant A). The goal of significance testing is to determine if there is sufficient evidence to reject the null hypothesis in favor of the alternative hypothesis (H₁), which claims a real difference exists.

• Example: H₀: "The click-through rate (CTR) for Model B is equal to the CTR for Model A."
• Purpose: Serves as a skeptical, falsifiable starting point for the test.

The alternative hypothesis is the assertion that contradicts the null hypothesis. It represents the effect the experiment is designed to detect. In production canary analysis, this is typically a one-sided hypothesis (e.g., Model B is better than Model A) rather than a two-sided test for any difference.

• Example (One-sided): H₁: "The CTR for Model B is greater than the CTR for Model A."
• Business Alignment: Defines the success metric and direction of improvement (e.g., higher accuracy, lower latency).

The p-value quantifies the probability of observing the experimental results, or something more extreme, assuming the null hypothesis is true. A low p-value indicates that the observed data is unlikely under the null hypothesis.

• Interpretation: A p-value of 0.03 means there is a 3% chance of seeing the observed difference if no real difference existed.
• Threshold (Alpha, α): Results are deemed statistically significant if the p-value falls below a pre-defined significance level, commonly α = 0.05. This is not the probability the null hypothesis is true.

The significance level (α) is the maximum acceptable probability of making a Type I error—falsely rejecting a true null hypothesis (a false positive). It is the threshold against which the p-value is compared.

• Standard Value: α = 0.05 (5% risk). More stringent fields may use α = 0.01.
• Trade-off: Lowering α reduces false positives but increases the risk of Type II errors (false negatives).
• Pre-Experiment Setting: Must be defined before data collection begins to avoid p-hacking.

Statistical power is the probability of correctly rejecting a false null hypothesis—detecting a real effect when it exists. It is the complement of the Type II error rate (β). Low power leads to inconclusive tests and missed opportunities to identify superior models.

• Factors Increasing Power: Larger sample size, larger true effect size, and higher significance level (α).
• Target: Industry standards often target 80% power (β = 0.20).
• Critical for Planning: Power analysis is used before a test to calculate the required sample size or minimum detectable effect.

A confidence interval provides a range of plausible values for the true effect size (e.g., the difference in conversion rates), with a stated level of confidence (e.g., 95%). It conveys both the magnitude and precision of the estimated effect, offering more information than a binary significant/not-significant result.

• Interpretation: A 95% CI for a lift of 2% being [0.5%, 3.5%] means we are 95% confident the true lift lies between 0.5% and 3.5%.
• Link to Significance: If a 95% confidence interval for the difference excludes zero, it is equivalent to a statistically significant result at α = 0.05.

The p-value is the probability of observing the measured difference between two groups (e.g., control vs. canary) if there is no actual underlying difference (the null hypothesis). A low p-value (typically < 0.05) provides evidence to reject the null hypothesis, suggesting the observed difference is statistically significant and not due to random chance.

• Interpretation: A p-value of 0.01 means there's a 1% chance the observed result occurred randomly.
• Thresholds: Common significance levels (alpha) are 0.05 or 0.01.
• Caution: A statistically significant result is not necessarily practically significant; the effect size must also be considered for business impact.

A confidence interval provides a range of plausible values for an estimated metric (e.g., conversion rate difference), calculated from sample data. A 95% confidence interval means that if the experiment were repeated many times, 95% of the calculated intervals would contain the true population parameter.

• Use in Canary Analysis: Instead of a binary significant/not-significant verdict, confidence intervals show the magnitude and uncertainty of the observed effect.
• Interpretation: If a 95% CI for a latency reduction is [10ms, 30ms], we are 95% confident the true reduction lies within that range.
• Relationship to P-Value: If a 95% confidence interval for a difference does not include zero, it corresponds to a p-value < 0.05.

Statistical power is the probability that a test will correctly reject a false null hypothesis—that is, detect a real effect when one exists. Low power increases the risk of a Type II error (false negative), where a beneficial model update is incorrectly deemed not significant.

• Factors Affecting Power: Sample size, effect size, and significance level (alpha).
• Canary Deployment Implication: Underpowered tests in early canary stages (with low traffic) may fail to detect meaningful regressions, allowing bugs to propagate. Automated Canary Analysis (ACA) tools often require minimum traffic volumes to ensure sufficient power.
• Target: Industry standards often aim for a power of 0.8 (80%).

The multiple testing problem arises when many statistical hypotheses are tested simultaneously (e.g., monitoring dozens of metrics during a canary), increasing the probability that at least one test will show a false positive (Type I error) purely by chance.

• Example: With 20 independent metrics and a 5% significance level, the chance of at least one false alarm is ~64%.
• Mitigation Strategies:
  • Bonferroni Correction: Divides the significance threshold (alpha) by the number of tests. Conservative but simple.
  • False Discovery Rate (FDR) Control: Methods like Benjamini-Hochberg that limit the proportion of false positives among declared discoveries.
  • Primary/Guardrail Metrics: Prioritizing a few key SLOs for the deployment verdict.

Effect size is a quantitative measure of the magnitude of the difference between two groups, independent of sample size. While statistical significance tells you if an effect exists, effect size tells you how large it is.

• Common Measures: Cohen's d (standardized mean difference), relative risk, absolute difference in conversion rates.
• Business Impact: A tiny latency improvement (e.g., 0.1ms) might be statistically significant with huge traffic but is operationally irrelevant. Canary analysis must consider minimum detectable effect (MDE) thresholds that align with business SLOs.
• Practical Significance: The combination of statistical significance and a meaningful effect size determines whether a canary should be promoted.

Sequential testing is an experimental design where data is evaluated as it arrives, allowing for early stopping once significance is reached or futility is determined. This contrasts with fixed-horizon testing that requires a predetermined sample size.

• Advantage in Canary Analysis: Enables faster promotion of clear winners or quicker rollback of severe regressions, optimizing the speed and safety of deployments.
• Methods: Includes Sequential Probability Ratio Test (SPRT) and Bayesian approaches.
• Consideration: Requires adjustments to maintain correct error rates when performing peeks at the data. Tools like Google's Optimize and Kayenta often implement sequential testing logic for automated canary analysis.