Bonferroni Correction

The correction adjusts the significance threshold by dividing the desired overall alpha level (α) by the number of independent hypotheses (m) being tested.

• Adjusted Alpha: α_new = α / m
• Example: For 20 variants tested against a control with a desired α of 0.05, each individual test must achieve a p-value < 0.0025 to reject the null hypothesis.
• Direct P-Value Adjustment: Alternatively, multiply each raw p-value by m, capping the result at 1.0, and compare against the original α.

The primary objective is to strictly limit the probability of making one or more Type I errors (false positives) across the entire family of comparisons.

• Strong Control: Guarantees FWER ≤ α for any configuration of true and false null hypotheses.
• Contrast with FDR: Unlike the False Discovery Rate, which controls the proportion of false positives, Bonferroni controls the occurrence of any false positive, making it suitable for high-stakes confirmatory trials.

The correction is known for being highly conservative, especially as the number of tests increases, which directly reduces statistical power.

• Increased Type II Errors: By making it harder to reject the null hypothesis, the probability of missing a true effect (false negative) increases.
• Independence Assumption: The correction is exact for independent tests but becomes overly conservative when test statistics are positively correlated, a common scenario in A/B testing where metrics are related.

In online controlled experiments, Bonferroni is applied when evaluating multiple goal metrics or treatment arms against a single control to avoid the peeking problem across metrics.

• Multiple Variants: If an A/B/n test has one control and four treatments, m=4 comparisons are made.
• Multiple Metrics: When monitoring a North Star Metric alongside several guardrail metrics, a correction ensures the overall chance of a false positive across all monitored metrics remains controlled.

Due to its conservatism, several less stringent alternatives are often preferred in large-scale experimentation platforms.

• Holm-Bonferroni Method: A step-down procedure that uniformly provides more power than the simple Bonferroni correction while still controlling FWER.
• Benjamini-Hochberg Procedure: Controls the False Discovery Rate (FDR) instead of FWER, offering a better balance between discovery and false positive control for exploratory analysis with thousands of metrics.

Consider a personalization model test evaluating click-through rate (CTR), conversion rate, and average order value (AOV) against a control.

• Scenario: 3 primary metrics are tested simultaneously.
• Calculation: α_new = 0.05 / 3 ≈ 0.0167.
• Interpretation: A raw p-value of 0.03 for the CTR lift would be considered statistically significant under standard rules but is not significant after the Bonferroni correction, preventing a premature conclusion.

The probability of making one or more Type I errors (false positives) when performing multiple simultaneous hypothesis tests. The Bonferroni correction directly controls this rate by adjusting the significance threshold. If you test 20 variants independently at α=0.05, the FWER balloons to approximately 64%, making the Bonferroni adjustment essential for maintaining experimental integrity in multivariate personalization tests.

The expected proportion of rejected null hypotheses that are actually true. Unlike the Bonferroni correction, which controls the family-wise error rate, FDR-controlling procedures like the Benjamini-Hochberg method offer a less conservative alternative. This is critical in large-scale experimentation platforms where testing thousands of metrics simultaneously requires balancing statistical rigor with the practical need to detect genuine effects without excessive false negatives.

Occurs when the null hypothesis is incorrectly rejected, leading to the conclusion that a variant has a significant effect when it does not. The Bonferroni correction directly mitigates this risk by dividing the significance level (α) by the number of comparisons. For example, testing 10 metrics at α=0.05 requires a corrected threshold of α=0.005 per test, dramatically reducing the chance of shipping a spurious personalization model to production.

A step-down sequential alternative to the simple Bonferroni correction that uniformly provides more statistical power while still controlling the family-wise error rate. It works by ordering p-values from smallest to largest and comparing each to a progressively less stringent threshold. This method is preferred in dynamic retail experimentation where the simple Bonferroni correction may be too conservative, causing missed opportunities for genuine personalization improvements.

The fundamental statistical challenge that the Bonferroni correction addresses. As the number of independent hypothesis tests increases, the probability of observing at least one statistically significant result purely by chance rises exponentially. In A/B testing infrastructure for AI, this manifests when evaluating multiple model variants, guardrail metrics, and customer segments simultaneously, requiring rigorous correction to prevent false discoveries from degrading the user experience.

A slightly less conservative alternative to the Bonferroni correction that assumes independence among tests. The adjusted alpha is calculated as 1-(1-α)^(1/m) rather than α/m. While mathematically more powerful when tests are truly independent, it is less commonly used in practice because the assumption of independence rarely holds in real-world A/B testing environments where metrics are often correlated. The Bonferroni remains the safer default.