False Discovery Rate (FDR)

The critical distinction between FDR and Family-Wise Error Rate (FWER) lies in their tolerance for errors:

• FWER (e.g., Bonferroni correction): Controls the probability of making one or more Type I errors. Extremely conservative, often leading to many missed discoveries in high-dimensional settings.
• FDR: Controls the expected proportion of false positives among discoveries. More liberal, allowing a controlled number of false leads to surface. In quantitative finance, where testing 10,000+ factors is common, FDR is preferred because it balances the risk of spurious alpha against the opportunity cost of discarding genuine predictive signals.

In quantitative finance, FDR is a critical safeguard against data snooping and p-hacking:

• Multiple Testing Problem: When backtesting thousands of candidate factors, some will appear significant purely by chance. Without correction, researchers will select these spurious signals for live trading.
• FDR Control: By setting a target FDR of 10%, a quant team accepts that up to 10% of their deployed factors may be false positives, while capturing 90% of genuine alpha.
• Deflated Sharpe Ratio: An extension of FDR thinking to strategy evaluation, adjusting Sharpe ratios for the expected maximum that would arise from multiple testing.
• Walk-Forward Validation: Combining FDR control with out-of-sample testing further reduces the risk of deploying noise as alpha.

FDR control relies on several assumptions that may be violated in practice:

• Independence: The BH procedure assumes tests are independent or exhibit positive regression dependency. Correlated factors (e.g., multiple momentum variants) can inflate the actual FDR beyond the nominal level.
• P-value Accuracy: FDR control is only as reliable as the p-values it adjusts. Model misspecification, heteroskedasticity, or non-normal return distributions can distort p-values.
• Publication Bias: In academic finance, the selective reporting of significant results distorts the true discovery rate across the literature.
• Adaptive Thresholds: The Benjamini-Yekutieli procedure provides a more conservative adjustment for arbitrary dependency structures, at the cost of reduced power.

The probability of making at least one Type I error across an entire family of hypothesis tests. Unlike FDR, which controls the proportion of false positives, FWER controls the probability of any false positive occurring. The classic Bonferroni correction is an FWER-controlling procedure that divides the significance threshold α by the number of tests m.

• Key distinction: FWER is far more conservative than FDR, making it suitable when even a single false positive is catastrophic (e.g., drug approval trials)
• In alpha research: FWER often eliminates too many true discoveries, making FDR the preferred framework for exploratory signal mining
• Common methods: Bonferroni, Holm-Bonferroni, and Hochberg step-up procedures

An extension of the Benjamini-Hochberg procedure that controls FDR under arbitrary dependence structures among test statistics. It applies a more conservative penalty by dividing the threshold by the harmonic sum H(m) = Σ(1/i) for i=1 to m.

• When to use: When alpha factors exhibit complex correlation structures, such as clustered signals within the same sector or asset class
• Trade-off: Lower statistical power than Benjamini-Hochberg in exchange for robustness to arbitrary dependencies
• Practical impact: In a universe of 1,000 tests, the penalty factor H(1000) ≈ 7.49, substantially raising the bar for significance

The FDR analogue of the p-value, representing the minimum FDR at which a given hypothesis would be rejected. A q-value of 0.03 means that if all tests with q ≤ 0.03 are declared significant, the expected proportion of false positives among them is 3%.

• Calculation: q(p(i)) = min{ p(k) × m / k } for all k ≥ i, using the Benjamini-Hochberg framework
• Interpretation: Unlike p-values, q-values directly answer "what proportion of my discoveries are likely false?"
• Storey's method: An alternative estimation approach using π₀, the estimated proportion of true null hypotheses, to improve power when many signals are real

A practical framework for penalizing backtested Sharpe Ratios based on the number of strategy variations explored during development. The haircut is proportional to the expected maximum of a set of random Sharpe Ratios.

• Rule of thumb: Expected maximum Sharpe ≈ σ × √(2 × ln(N)), where N is the number of independent trials and σ is the standard deviation
• Example: Testing 100 uncorrelated strategies with σ=1 yields an expected maximum Sharpe of ~3.03 purely by chance
• Implementation: Divide the reported Sharpe by this expected maximum to obtain a "haircut" Sharpe that better reflects true skill
• Integration with FDR: Both frameworks address the same underlying problem—distinguishing genuine alpha from statistical artifacts in high-dimensional search spaces