Null Hypothesis

The null hypothesis represents the status quo or the assumption that any observed difference is due to random chance. In A/B testing, H₀ typically states that the conversion rate of variant A equals variant B. The experimenter's burden is to gather sufficient evidence to reject this assumption, not to prove the alternative hypothesis directly. This framework protects against confirmation bias by forcing a structured, evidence-based approach to decision-making.

Mathematically, the null hypothesis is expressed as an equality statement:

• H₀: μ₁ = μ₂ (no difference between population means)
• H₀: ρ = 0 (no correlation between variables)
• H₀: β = 0 (no effect of a feature in regression)

The alternative hypothesis (H₁ or Hₐ) is the complement, representing the presence of an effect. This binary framing enables the calculation of p-values and test statistics under a known sampling distribution.

A critical nuance: experimenters never accept the null hypothesis. They either:

• Reject H₀: Sufficient evidence exists to conclude a statistically significant effect
• Fail to reject H₀: Insufficient evidence to conclude an effect, but this does not prove the null is true

This distinction prevents the logical fallacy of arguing from ignorance. A non-significant result may indicate a true null, insufficient statistical power, or a poorly designed experiment.

The significance level (α) is the pre-specified probability threshold for rejecting H₀ when it is actually true. Common values:

• α = 0.05: 5% risk of a false positive
• α = 0.01: 1% risk for high-stakes decisions

When a p-value falls below α, the result is declared statistically significant. This directly controls the Type I error rate—the probability of claiming a winning variant when no real difference exists. Setting α too high inflates false discoveries; setting it too low increases Type II errors.

The null hypothesis embodies the principle of parsimony: the simplest explanation (no effect) is preferred until evidence demands otherwise. In frequentist inference, H₀ provides the sampling distribution used to calculate how extreme an observed result is. Without a clearly defined null, there is no baseline for measuring effect size or determining if a metric movement is attributable to the treatment rather than random variation in the data.

Rejecting the null hypothesis only confirms that an effect likely exists, not that it is practically meaningful. With large sample sizes, even trivially small differences can produce statistically significant p-values. Experimenters must evaluate:

• Effect size: The magnitude of the difference (e.g., Cohen's d)
• Confidence intervals: The precision of the estimate
• Business impact: Whether the lift justifies engineering cost

A significant result with a 0.01% lift may be statistically valid but operationally irrelevant.

The probability of observing a test statistic at least as extreme as the one calculated, assuming the null hypothesis is true. It is a continuous measure of compatibility between the observed data and the null hypothesis.

• Common threshold: p < 0.05 for rejecting the null
• Misinterpretation risk: A p-value is not the probability that the null hypothesis is false
• Calculation: Derived from the sampling distribution of the test statistic (e.g., t-distribution)

A false positive error that occurs when the null hypothesis is incorrectly rejected. The experimenter concludes a variant has a significant effect when it actually does not.

• Significance level (α): The pre-specified probability of committing a Type I error, typically set at 0.05
• Impact: Leads to shipping features or models that provide no real improvement
• Control: Managed by setting a strict alpha threshold and adjusting for multiple comparisons

A false negative error that occurs when the null hypothesis is incorrectly retained. A genuine improvement from a treatment variant is missed due to insufficient evidence.

• Beta (β): The probability of committing a Type II error
• Statistical Power (1-β): The probability of correctly rejecting a false null hypothesis
• Common cause: Insufficient sample size or too short an experiment duration
• Mitigation: Conduct a power analysis before launching the experiment

A range of values, derived from sample data, that is likely to contain the true population parameter with a specified probability (typically 95%).

• Interpretation: If the experiment were repeated many times, 95% of the calculated intervals would contain the true effect
• Practical use: Provides a measure of precision and uncertainty around the point estimate of the lift
• Decision rule: If the interval for a metric excludes zero, the result is statistically significant at the corresponding alpha level

The probability that a statistical test will correctly reject a false null hypothesis. It represents the experiment's sensitivity to detect a true effect if one exists.

• Target: Conventionally set at 80% (0.80)
• Key drivers: Sample size, effect size, and significance level (α)
• Trade-off: Increasing power requires more traffic or a longer experiment duration
• Calculation: Performed via power analysis using the minimum detectable effect (MDE)

A quantitative measure of the magnitude of the difference between two groups, providing a standardized assessment of practical significance independent of sample size.

• Cohen's d: Standardized mean difference for continuous metrics
• Relative lift: Percentage change, common for conversion rates and revenue
• Why it matters: A statistically significant result with a negligible effect size may not be worth the engineering cost of deployment