Empirical Bayes Shrinkage

Empirical Bayes operates on a hierarchical model with two levels:

• Prior distribution: A parametric distribution (e.g., inverse gamma, log-normal) describing how dispersion or fold change varies across all genes in the experiment.
• Likelihood: A gene-specific sampling distribution (e.g., negative binomial for counts) given the true parameter value.

By combining these levels via Bayes' theorem, the method computes a posterior estimate for each gene that optimally weights the gene's own data against the global prior. Genes with sparse or noisy data are pulled more strongly toward the prior mean, while genes with abundant, consistent data retain their original estimates.

In tools like DESeq2 and limma, shrinkage is not toward a single global mean but toward a fitted mean-variance trend line:

• The prior mean is a smooth function of expression level, estimated empirically from all genes.
• Low-count genes, which exhibit extreme and unreliable raw dispersion estimates, are shrunken toward the trend value predicted for their expression level.
• High-count genes with stable estimates are largely unshrunken.

This intensity-dependent shrinkage ensures that the prior adapts to the data structure rather than imposing a one-size-fits-all correction. The result is dramatically improved ranking and visualization of fold changes.

The core computation yields a posterior estimate that is a precision-weighted average:

• Gene-specific estimate: The raw log2 fold change or dispersion calculated from that gene's counts alone.
• Prior estimate: The value predicted by the empirical prior distribution.
• Weighting factor: Determined by the relative precision of the gene-specific estimate versus the prior precision.

Genes with low precision (high uncertainty) receive more weight on the prior, pulling them toward the center. This moderated estimate has lower mean squared error than the raw estimate, reducing false positives from extreme but unreliable values while preserving true large effects.

Shrinkage directly improves statistical power and error control:

• By stabilizing dispersion estimates, the variance of test statistics is reduced, making the Wald test or moderated t-test more reliable.
• Genes with artificially low dispersion due to sampling noise are prevented from achieving spuriously significant p-values.
• The number of false positives at any given False Discovery Rate (FDR) threshold is reduced.
• True differentially expressed genes with borderline effect sizes become detectable because the noise floor is lowered.

This property is especially critical in small-sample experiments where per-gene variance estimates are inherently unstable.

A key theoretical property of empirical Bayes shrinkage is asymptotic consistency:

• As sample size increases, the influence of the prior diminishes, and the posterior estimate converges to the gene-specific maximum likelihood estimate.
• The prior is estimated from the data itself (hence 'empirical' Bayes), so it adapts to the experimental scale.
• In large experiments, shrinkage becomes negligible for most genes, preserving the ability to detect subtle effects.

This ensures the method does not introduce systematic bias in well-powered studies while providing critical stabilization in underpowered ones—a property that distinguishes it from arbitrary regularization techniques.

Modern implementations like apeglm (adaptive t prior) extend basic shrinkage:

• Uses a heavy-tailed t-distribution as the prior for log fold changes rather than a normal distribution.
• The heavy tails allow large true effects to escape shrinkage, preventing attenuation of genuinely large fold changes.
• The prior parameters are estimated via maximum marginal likelihood, adapting to the observed distribution of effects.
• Produces posterior probability and local false sign rate (lfsr) for each gene, offering more nuanced inference than p-values alone.

This approach is particularly valuable when the true effect size distribution is sparse—most genes have no change, but a few have very large changes.