DESeq2 Normalization

The core normalization step calculates a size factor for each sample to account for sequencing depth and compositional bias. The process is:

• A pseudo-reference sample is created by taking the geometric mean of each gene across all samples.
• For each sample, the ratio of its raw counts to the pseudo-reference is computed for every gene.
• The median of these ratios for a sample becomes its size factor. This method is robust to the influence of a few highly and differentially expressed genes, as the median is insensitive to outliers, unlike a total sum normalization.

DESeq2 models raw counts with a Negative Binomial (NB) distribution, which is a generalization of the Poisson distribution that accounts for overdispersion—the greater-than-expected variability seen in biological data. The model estimates:

• A gene-specific dispersion parameter, capturing biological variability.
• A fitted mean-dispersion relationship, sharing information across genes with similar expression strength to stabilize estimates, especially for genes with low counts. This approach prevents false positives from genes with high variance due to biological noise rather than a true treatment effect.

For visualization and clustering tasks like PCA or heatmaps, DESeq2 offers a Variance Stabilizing Transformation (VST). This transforms the normalized count data so that the variance becomes approximately independent of the mean. Key properties:

• It renders the data homoskedastic, a core assumption of many linear methods.
• It is a more sophisticated alternative to a simple log2(counts + 1) transformation, which can inflate the variance of low-count genes.
• The VST is derived from the fitted dispersion-mean relationship, making it data-driven rather than a fixed function.

DESeq2 automatically flags individual sample-gene combinations that are potential outliers using Cook's distance, a measure of influence in linear regression. A gene count is flagged if:

• Its Cook's distance exceeds a threshold based on the F-distribution.
• The count is an extreme outlier relative to the fitted NB model for that gene. These counts can be optionally replaced with a trimmed mean or the gene can be filtered from results, preventing a single spurious count from driving a false differential expression call.

After normalization and dispersion estimation, differential expression is assessed using a generalized linear model. DESeq2 provides two tests:

• Wald test: Efficiently tests a single coefficient (e.g., the difference between two conditions) in the fitted model.
• Likelihood Ratio Test (LRT) : Compares a full model to a reduced model to test multiple coefficients simultaneously, useful for time-series or multi-factor designs. Both tests generate p-values, which are then corrected for multiple testing using the Benjamini-Hochberg procedure to control the false discovery rate.

To increase statistical power, DESeq2 internally performs independent filtering on genes with very low mean normalized counts. The rationale is that genes with extremely low expression levels have little to no chance of being declared statistically significant, so removing them before p-value adjustment reduces the burden of multiple testing correction. The algorithm:

• Finds the optimal quantile of mean counts to filter that maximizes the number of rejections.
• This filtering is independent of the test statistic under the null hypothesis, preserving type I error control. Users see more adjusted p-values below their significance threshold without compromising statistical validity.

DESeq2 models read count data using a negative binomial (NB) distribution, which is a generalization of the Poisson distribution that accounts for overdispersion—the phenomenon where the variance of counts exceeds the mean. The NB distribution has two parameters: the mean (linked to expression level) and a dispersion parameter (alpha). DESeq2 shrinks dispersion estimates toward a trended curve to stabilize estimates, especially for genes with low counts.

A data transformation applied downstream of DESeq2's size factor estimation to render the variance independent of the mean, producing homoskedastic data. This is critical for exploratory techniques like Principal Component Analysis (PCA) or hierarchical clustering, which assume constant variance. The VST effectively log2-transforms the normalized counts while moderating the high variance of low-count genes, preventing them from dominating distance metrics.

An alternative normalization method implemented in the edgeR package. TMM calculates a scaling factor based on the weighted trimmed mean of log expression ratios between a sample and a reference. It trims genes with extreme log-fold changes (often 30% by M-value and 5% by expression level) under the assumption that most genes are not differentially expressed. This contrasts with DESeq2's median-of-ratios approach, which uses a pseudo-reference.

An extension of the ComBat empirical Bayes batch correction framework specifically designed for RNA-seq count data. Unlike standard ComBat, which assumes normally distributed data, ComBat-Seq uses a negative binomial regression model to adjust for known batch effects while preserving the integer nature of counts. It estimates additive batch effects on the log scale and applies a shrinkage step, making it a direct complement to DESeq2's normalization for multi-batch experiments.

A normalization strategy that uses negative control genes—genes known a priori not to be differentially expressed with respect to the biological condition—or replicate samples to estimate factors of unwanted technical variation. RUVSeq decomposes the expression matrix using factor analysis, identifies the unwanted factors, and regresses them out. This is particularly powerful when batch effects are confounded with the condition of interest and require explicit modeling beyond simple size factor scaling.