Bonferroni Correction

The Bonferroni correction operates on a simple principle: divide the alpha level by the number of comparisons.

• If testing 10 hypotheses at α = 0.05, the corrected threshold becomes α_corrected = 0.05 / 10 = 0.005
• Each individual test must achieve p < 0.005 to reject the null hypothesis
• This directly controls the family-wise error rate (FWER) — the probability of making at least one Type I error across all tests
• The method makes no assumptions about the dependence structure among tests, making it universally applicable

The correction is derived from Boole's inequality, which states that the probability of a union of events is less than or equal to the sum of their individual probabilities.

• For m independent tests: P(at least one false positive) ≤ m × α
• Setting m × α = 0.05 yields the Bonferroni adjustment: α / m
• This guarantees strong control of the FWER at the desired level
• The inequality holds regardless of correlation between tests, making the method robust but inherently conservative

The Bonferroni correction is widely recognized as overly conservative, especially with large numbers of comparisons.

• As the number of tests increases, the corrected threshold becomes extremely stringent
• This dramatically reduces statistical power — the ability to detect true effects
• For 100 tests at α = 0.05, the threshold drops to 0.0005, potentially missing genuine diagnostic signals
• In medical imaging studies with thousands of voxel-wise comparisons, the correction may be impractically strict
• Alternative methods like the Benjamini-Hochberg procedure control the false discovery rate instead, offering greater power

In clinical validation of diagnostic AI, the Bonferroni correction is applied when evaluating multiple endpoints or subgroup analyses.

• When comparing AI performance across multiple anatomical regions, each comparison inflates the Type I error risk
• A study evaluating sensitivity for 5 different lesion types would use α = 0.01 per test
• Regulatory bodies like the FDA often expect multiplicity adjustments in pivotal trials with co-primary endpoints
• The correction is also used in reader studies when comparing multiple radiologist-AI interaction scenarios
• Failing to adjust for multiplicity can lead to spurious claims of diagnostic superiority

Several modifications address the Bonferroni correction's conservatism while maintaining FWER control.

• Holm-Bonferroni method: A step-down procedure that sequentially rejects hypotheses, uniformly more powerful than the standard Bonferroni
• Šidák correction: Uses α_S = 1 - (1 - α)^(1/m), slightly less conservative when tests are independent
• Hochberg's step-up procedure: Offers greater power for positively correlated tests
• Permutation testing: Computes an empirical null distribution by shuffling labels, accounting for the actual correlation structure
• In genomic and radiomic studies with thousands of features, the Benjamini-Hochberg FDR control is strongly preferred over FWER methods

Clinical validation studies must transparently document multiplicity adjustments to meet regulatory and publication standards.

• Clearly state the number of comparisons and the rationale for the chosen correction method
• Report both unadjusted and adjusted p-values to allow independent assessment
• Specify whether the analysis was pre-specified or exploratory — post-hoc multiplicity adjustments carry less evidentiary weight
• The CONSORT-AI and STARD-AI reporting guidelines emphasize explicit documentation of multiplicity strategies
• For FDA submissions, the statistical analysis plan must prospectively define the multiplicity adjustment approach to maintain trial integrity

The probability of making one or more Type I errors across a family of hypothesis tests. The Bonferroni correction directly controls FWER at a specified alpha level (e.g., 0.05) by dividing the threshold by the number of comparisons.

• Conservative approach: Guarantees strong control regardless of dependency structure
• Trade-off: Reduces statistical power, increasing Type II error risk
• Clinical context: Critical when evaluating multiple imaging endpoints where a single false positive could lead to an invalid regulatory claim

A step-down sequential rejection procedure that uniformly improves upon the Bonferroni correction while maintaining strong FWER control. It orders p-values from smallest to largest and applies progressively less stringent thresholds.

• Always more powerful than the standard Bonferroni correction
• Procedure: Compare smallest p-value to α/m, next to α/(m-1), continuing until a non-rejection occurs
• Clinical advantage: Maintains the same rigorous error control while increasing the chance of detecting true diagnostic effects

The false positive conclusion—incorrectly rejecting a true null hypothesis. In diagnostic AI validation, this means claiming a model detects a condition when no real effect exists.

• Consequence in SaMD: A Type I error could lead to FDA clearance of an ineffective diagnostic tool
• Bonferroni's role: Directly controls the probability of committing any Type I error across multiple endpoints
• Relationship: FWER = P(at least one Type I error) across all tests in the family

A slightly less conservative alternative to Bonferroni that assumes independence among tests. The adjusted alpha is calculated as 1 - (1 - α)^(1/m) rather than α/m.

• Mathematical basis: Derived from the probability of no Type I errors across independent tests
• Limitation: Assumes independence, which rarely holds in correlated diagnostic endpoints
• Practical note: The difference from Bonferroni is negligible for small alpha values, making Bonferroni the safer default in clinical settings