DESeq2

DESeq2 models raw read counts using a negative binomial distribution, which naturally accounts for the overdispersion inherent in RNA-seq data. Unlike a Poisson distribution, which assumes variance equals the mean, the negative binomial allows variance to exceed the mean, capturing both technical shot noise and biological variability. The model parameterizes the variance as a function of the mean with a gene-specific dispersion parameter, enabling accurate hypothesis testing even for genes with low counts.

A defining innovation of DESeq2 is its use of empirical Bayes shrinkage for dispersion estimation. Raw dispersion estimates are highly unstable for genes with few counts. DESeq2 addresses this by:

• Fitting a parametric curve to the observed dispersion-mean relationship across all genes
• Shrinking individual gene dispersions toward this fitted trend
• Applying stronger shrinkage when counts are low and weaker shrinkage when counts are high This information-borrowing strategy stabilizes variance estimates and dramatically reduces false positives.

DESeq2 applies adaptive shrinkage to log2 fold change estimates, a step distinct from dispersion shrinkage. Using a zero-centered Normal prior whose width is estimated from the data, the algorithm shrinks large fold changes from genes with high dispersion or low counts toward zero. This produces conservative, reproducible effect size estimates that are directly suitable for ranking genes by biological significance rather than statistical significance alone. The lfcShrink() function implements this with options for the apeglm and ashr shrinkage estimators.

DESeq2 employs the Wald test to assess whether estimated log2 fold changes differ significantly from zero. For each gene, the test computes:

• The maximum likelihood estimate of the coefficient
• Its standard error from the model's information matrix
• A z-statistic and corresponding p-value For multi-factor designs, DESeq2 also supports likelihood ratio tests (LRT), which compare full and reduced model fits to identify genes that respond to any level of a categorical variable, making it ideal for time-series or multi-condition experiments.

DESeq2 integrates independent filtering to increase detection power. Before multiple testing correction, genes with very low mean normalized counts are filtered out, as they provide insufficient evidence for differential expression. This filtering is performed independently of the test statistic under the null hypothesis, so it does not inflate the type I error rate. By reducing the number of tests requiring correction, the Benjamini-Hochberg procedure operates on a smaller set, yielding more significant adjusted p-values for genes with sufficient signal.

DESeq2 provides a variance stabilizing transformation (VST) and a regularized log transformation (rlog) for downstream visualization. These transformations produce data where the variance is approximately independent of the mean, making them suitable for:

• Principal component analysis (PCA) to visualize sample clustering
• Heatmaps of gene expression patterns
• Machine learning inputs requiring homoskedastic data The VST is faster than rlog for large datasets and is the recommended default for exploratory analysis.