Negative Binomial Distribution

The defining feature that makes the negative binomial essential for genomics. Unlike the Poisson distribution, which assumes the variance equals the mean, the negative binomial introduces a dispersion parameter that allows the variance to exceed the mean.

• Biological reality: Gene expression varies between replicate samples due to natural biological fluctuations, not just sampling noise
• Technical reality: Library preparation and sequencing introduce additional variability
• Consequence: Using Poisson models on RNA-seq data produces dramatically inflated false positive rates because the model underestimates the true variability

The negative binomial distribution models the probability of observing a specific count given a mean expression level and a dispersion parameter.

For RNA-seq, the parameterization used by tools like DESeq2 and edgeR is:

• Mean (μ): The expected normalized count for a gene in a given condition
• Dispersion (α): Controls how much the variance exceeds the mean
• Variance function: Var = μ + α·μ²

When α → 0, the distribution converges to Poisson. Larger α values indicate stronger overdispersion, which is typical for genes with high biological variability.

The negative binomial can be understood as a gamma-Poisson mixture — a Poisson distribution where the rate parameter itself follows a gamma distribution.

• Poisson: Assumes a fixed, constant expression level across all replicates
• Negative Binomial: Acknowledges that the true expression level varies from sample to sample according to a gamma distribution
• Practical implication: This hierarchical structure naturally produces heavier tails than Poisson, accommodating the occasional outlier counts observed in real RNA-seq experiments without flagging them as differentially expressed

Accurate dispersion estimation is the critical computational challenge. Individual gene dispersion estimates are unstable for low-count genes, so modern tools employ shrinkage:

• DESeq2: Uses empirical Bayes shrinkage, assuming dispersion follows a log-normal prior distribution across genes of similar expression strength
• edgeR: Estimates a common dispersion across all genes, then a trended dispersion as a function of expression level, and finally a tagwise dispersion with empirical Bayes moderation
• Benefit: Information is borrowed across genes, producing more stable estimates and better Type I error control, especially for genes with low read counts

A fundamental property of RNA-seq count data is that the variance grows quadratically with the mean. The negative binomial captures this through its variance function.

• Low-count genes: Variance is dominated by Poisson-like sampling noise (Var ≈ μ)
• High-count genes: Biological variability dominates (Var ≈ α·μ²)
• Diagnostic use: Mean-dispersion plots are standard quality control tools to verify that the chosen model appropriately captures the data's variance structure

This relationship is why simple log-transformations of count data are insufficient — they fail to properly stabilize variance across the full dynamic range of expression.

The negative binomial distribution is embedded within a generalized linear model (GLM) framework in modern differential expression tools.

• Link function: A log link connects the linear predictor to the expected count: log(μ) = Xβ
• Design matrix (X): Encodes experimental conditions, batch effects, and covariates
• Coefficients (β): Represent log2 fold changes between conditions
• Hypothesis testing: Wald tests or likelihood ratio tests assess whether coefficients differ significantly from zero

This GLM structure enables complex experimental designs with multiple factors, continuous covariates, and interaction terms — far beyond simple two-group comparisons.

The phenomenon where the variance of count data is greater than the mean, violating the Poisson distribution's assumption of equality. In RNA-seq, overdispersion arises from biological variability between replicate samples and technical noise during library preparation. The negative binomial distribution explicitly models this extra variability through a dispersion parameter, preventing the false positives that would occur if a simpler Poisson model were used on highly variable biological data.

The process of quantifying the gene-specific biological variability around the expected mean expression level. In negative binomial models, the dispersion parameter (often denoted as α or φ) captures how much the true counts vary between biological replicates. Accurate estimation is critical: underestimated dispersion leads to false positives, while overestimated dispersion reduces statistical power. DESeq2 and edgeR use shrinkage estimators to stabilize these estimates across genes with similar expression levels.

The Poisson distribution assumes that the variance equals the mean, making it inadequate for RNA-seq data where biological replicates show greater variability. The negative binomial distribution extends the Poisson by adding a dispersion parameter that allows the variance to exceed the mean: Variance = Mean + Dispersion × Mean². This flexibility makes it the standard model for count-based differential expression tools, accurately capturing the overdispersion inherent in high-throughput sequencing experiments.

A flexible extension of ordinary linear regression that accommodates non-normal response distributions like the negative binomial. In differential expression analysis, GLMs allow researchers to model complex experimental designs with multiple factors and covariates simultaneously. The model uses a log link function to relate the expected counts to the linear predictor, enabling the testing of specific contrasts while controlling for batch effects, patient demographics, or other confounding variables.