Null Hypothesis

The null hypothesis is the default or status quo assumption that any observed difference in an experiment is due to random chance, not a systematic effect. It is a precise, testable statement about a population parameter (e.g., 'The mean click-through rate for variant A equals the mean for variant B'). The burden of proof lies on the alternative hypothesis (H₁) to provide sufficient evidence for rejection.

• Purpose: Serves as a skeptical baseline that experimental data must overcome.
• Example: In an A/B test for a new recommendation algorithm, H₀ states: 'The new algorithm does not increase average user engagement time compared to the old algorithm.'

A properly formulated null hypothesis must be mathematically falsifiable. It makes a specific claim about a population parameter (like a mean, proportion, or variance) that can be contradicted by sample data. Vague statements cannot be tested.

• Key Property: It is structured for potential rejection, not proof. You can never 'accept' or 'prove' H₀; you can only fail to reject it based on insufficient evidence.
• Statistical Model: It defines the expected distribution of the test statistic under the assumption of no effect (e.g., a t-distribution centered at zero for a difference in means).

The p-value is calculated directly under the assumption that H₀ is true. It represents the probability of observing data as extreme as, or more extreme than, the sample results, assuming the null hypothesis is correct. A small p-value indicates that the observed data is unlikely under H₀, leading to its rejection in favor of H₁.

• Interpretation: A p-value of 0.03 means there is a 3% chance of seeing the observed effect (or larger) if H₀ were true.
• Threshold: The pre-defined significance level (alpha, α)—commonly 0.05—is the threshold for rejecting H₀. If p ≤ α, H₀ is rejected.

Hypothesis testing errors are defined in relation to the truth of H₀.

• Type I Error (False Positive): Incorrectly rejecting a true null hypothesis. The probability of this error is controlled by the significance level (α).
• Type II Error (False Negative): Failing to reject a false null hypothesis. The probability of this error is denoted by beta (β). Statistical power is 1 - β, the probability of correctly rejecting a false H₀.

These error trade-offs are fundamental to experiment design, determining required sample sizes via power analysis.

While often an assertion of equality (e.g., μ₁ = μ₂), H₀ can also be formulated as a statement of 'no worse than' or using an inequality for one-sided tests.

• Two-Sided H₀: μ₁ = μ₂ (Tests for any difference)
• One-Sided H₀: μ₁ ≤ μ₂ (Tests specifically for an increase)

The choice between one-sided and two-sided tests must be made a priori, based on the research question, as it affects the p-value calculation and interpretation.

In A/B testing frameworks, the null hypothesis is the engine of decision-making. It allows the translation of a business question ('Does this new UI improve conversion?') into a statistical procedure.

• Assignment & Measurement: Users are randomly assigned to control (A) and treatment (B) groups. A metric (e.g., conversion rate) is measured for both.
• Test Execution: A statistical test (e.g., a chi-squared test for proportions, a t-test for means) computes a test statistic and corresponding p-value under H₀.
• Decision Rule: If p-value ≤ α, reject H₀ and conclude a statistically significant difference exists. The experiment then shifts to analyzing the estimated effect size and its confidence interval for practical significance.

A result is deemed statistically significant if the observed data is sufficiently unlikely under the assumption that the null hypothesis is true. This is formally determined by comparing a calculated p-value to a pre-defined significance level (alpha), commonly set at 0.05. Achieving statistical significance provides evidence to reject the null hypothesis, suggesting the observed effect (e.g., a model performance improvement) is real and not due to random chance.

The p-value quantifies the strength of evidence against the null hypothesis. It is the probability of observing a test statistic at least as extreme as the one calculated from your sample data, assuming the null hypothesis is true.

• A small p-value (e.g., < 0.05) indicates the observed data is improbable under the null, leading to its rejection.
• A large p-value suggests the data is compatible with the null hypothesis, so it is not rejected. Crucially, the p-value is not the probability that the null hypothesis is true or false.

Hypothesis testing involves two fundamental error types tied to the null hypothesis.

• Type I Error (False Positive): Incorrectly rejecting a true null hypothesis. The probability of this is controlled by the significance level (alpha).
• Type II Error (False Negative): Failing to reject a false null hypothesis. The probability of this is denoted by beta. The power of a test is (1 - beta), the probability of correctly rejecting a false null. Experiment design involves balancing these risks.

The alternative hypothesis (H1 or Ha) is the statement that directly contradicts the null hypothesis. It represents the effect or difference the experiment is designed to detect.

• In an A/B test comparing Model A and Model B, if the null is 'no difference in accuracy,' the alternative is 'there is a difference in accuracy.'
• It can be one-sided (specifying a direction, e.g., 'Model B is more accurate') or two-sided (specifying any difference). The test's statistical machinery evaluates evidence in favor of the alternative.

Statistical power is the probability that a test will correctly reject a false null hypothesis. It is the test's sensitivity to detect a true effect.

• High power (e.g., 0.8 or 80%) is desirable to avoid Type II Errors.
• Power depends on:
  • Sample Size: Larger samples increase power.
  • Effect Size: Larger true differences are easier to detect.
  • Significance Level (Alpha): A higher alpha increases power but also increases Type I Error risk. Calculating required sample size before an experiment ensures adequate power.

A confidence interval provides a range of plausible values for an unknown population parameter (e.g., the true difference in model performance). A 95% confidence interval means that if the experiment were repeated many times, 95% of such intervals would contain the true parameter value.

• Direct Relationship to Null Hypothesis: If a 95% confidence interval for a difference does not include zero, it corresponds to rejecting the null hypothesis at the 5% significance level.
• It conveys both the estimated effect size and the precision of the estimate, offering more information than a binary significance test.