Multiple Testing Correction

The simplest and most stringent family-wise error rate (FWER) control method. Divides the significance threshold α by the number of tests m.

• Adjusted threshold: α / m
• Extremely conservative: Controls probability of any false positive
• High false negative rate: Dramatically reduces statistical power in genomic contexts
• Best for: Small numbers of tests or when false positives are catastrophic

Example: Testing 20,000 genes at α=0.05 yields a threshold of 2.5×10⁻⁶.

The standard False Discovery Rate (FDR) control method in genomics. Controls the expected proportion of false positives among all discoveries rather than the probability of any single error.

• Step-up procedure: Ranks raw p-values from smallest to largest
• Adaptive threshold: Compares each pᵢ to (i/m)×α
• More power: Tolerates a controlled number of false positives
• Default in DESeq2 and edgeR: Industry standard for RNA-seq

FDR of 5% means ~5% of your significant gene list are expected to be false discoveries.

The fundamental tension in multiple testing correction is between stringency and statistical power.

• FWER (Bonferroni): "I want zero false positives, even if I miss real signals"
• FDR (Benjamini-Hochberg): "I accept some false positives to capture more true discoveries"
• Genomics default: FDR at 5% or 10% — the field prioritizes discovery over absolute certainty
• Clinical diagnostics: May require FWER control when false positives carry patient harm risk

Choosing the wrong method can mean the difference between a failed experiment and a breakthrough biomarker panel.

An extension of the Benjamini-Hochberg framework that estimates the positive false discovery rate (pFDR) and provides a more interpretable measure of significance.

• q-value: The minimum FDR at which a test would be called significant
• π₀ estimation: Estimates the proportion of true null hypotheses from the data
• More adaptive: Accounts for the actual null distribution shape
• Implementation: Available in the qvalue R/Bioconductor package

A q-value of 0.05 means: among features with q ≤ 0.05, approximately 5% are expected to be false positives.

A modern FDR control method that increases power by using an independent covariate to weight hypotheses before correction.

• Covariate example: Base mean expression level — lowly expressed genes get different weights
• Maintains FDR control: Proven to control FDR while increasing discoveries
• Data-driven: Learns optimal weighting function from the data
• Implementation: IHW R/Bioconductor package, integrates with DESeq2

IHW can increase the number of significant genes by 10-50% compared to standard Benjamini-Hochberg, particularly when power varies systematically across tests.

A non-parametric approach that empirically estimates the null distribution by repeatedly shuffling sample labels and recalculating test statistics.

• No distributional assumptions: Works when parametric assumptions fail
• Computationally intensive: Requires thousands of permutations
• Westfall-Young procedure: Controls FWER using permutation-based step-down adjustment
• Common in GSEA: Gene Set Enrichment Analysis uses permutation to assess enrichment significance

Particularly valuable when the theoretical null distribution is unknown or when test statistics are correlated across genes.