Dispersion Estimation

The dispersion parameter (α) captures the extra-Poisson variation inherent in RNA-seq count data. When α > 0, the variance exceeds the mean, following the negative binomial relationship: Variance = μ + αμ². Without modeling overdispersion, standard Poisson tests would produce dramatically inflated Type I error rates, as biological replicates show far more variability than technical replicates alone would predict.

Dispersion is not constant across expression levels. A fundamental property observed in real data is the inverse relationship between mean expression and dispersion:

• Highly expressed genes exhibit low dispersion (precise estimates)
• Lowly expressed genes show high, unstable dispersion This dependence must be modeled explicitly, typically by fitting a parametric curve (e.g., local regression) to the mean-dispersion trend across all genes.

Raw dispersion estimates for individual genes are unreliable when replicate counts are low. Empirical Bayes shrinkage borrows information across the entire dataset by shrinking gene-specific estimates toward a fitted prior distribution. This produces moderated dispersion estimates that are more stable than raw values, particularly for genes with few replicates or low counts. DESeq2 implements this via an apeglm or ashr shrinkage estimator.

Early methods assumed a common dispersion value for all genes—computationally simple but biologically unrealistic. Modern approaches estimate gene-wise dispersion (tagwise dispersion in edgeR) to capture true biological heterogeneity. The trade-off is that gene-wise estimates require sufficient replicates. When replicates are scarce, a trended dispersion approach models dispersion as a smooth function of expression level, balancing flexibility with stability.

edgeR estimates the dispersion parameter using Cox-Reid profile-adjusted likelihood, which conditions out the abundance parameter to focus estimation solely on dispersion. This approach reduces bias compared to maximum likelihood estimation, especially with small sample sizes. The method iteratively optimizes a negative binomial generalized linear model to find the dispersion value that best explains the observed count variation across replicates.

Outlier counts can severely distort dispersion estimates. Robust estimation methods in DESeq2 and edgeR-robust down-weight or exclude genes with extreme residuals during model fitting. Key techniques include:

• Huber's M-estimation for robust regression
• Residual deviance monitoring to flag hyper-variable genes
• Cook's distance to identify influential observations This prevents a small number of aberrant counts from inflating the dispersion prior and masking true differential expression signals.