DeLong Test

The DeLong test is specifically designed to compare Receiver Operating Characteristic (ROC) curves that are correlated—meaning they are derived from the same set of cases. This is the standard scenario when comparing two diagnostic models or two imaging modalities on a single patient cohort. Standard tests for independent samples would produce inflated Type I error rates in this context. The test accounts for the covariance structure between the Area Under the Curve (AUC) estimates.

Unlike parametric methods that assume a specific distribution (e.g., binormal), the DeLong test is non-parametric. It uses the structural components of the Mann-Whitney U-statistic to estimate the variance-covariance matrix of the AUCs. This makes it robust to violations of normality and particularly suitable for the often non-Gaussian distributions of prediction scores in machine learning models. The method relies on asymptotic theory, making it valid for sufficiently large sample sizes.

The core mathematical innovation of the DeLong test is its method for estimating the variance-covariance matrix of multiple AUC estimators. It computes a covariance for every pair of models being compared by evaluating the variability contributed by each individual case. This matrix is then used to construct a chi-squared test statistic to evaluate the null hypothesis that all AUCs are equal. The test statistic is calculated as:

• χ² = (Lθ)' [LSL']⁻¹ (Lθ) where θ is the vector of AUCs and S is the covariance matrix.

The DeLong test can be applied in two modes:

• Omnibus Test: Evaluates the global null hypothesis that all k models have identical AUCs. A significant result indicates at least one model differs.
• Pairwise Post-hoc: If the omnibus test is significant, or if only two models are compared, the test provides a direct contrast between two specific AUCs. This yields a z-statistic and a p-value for the difference between two models, allowing for direct head-to-head comparisons of diagnostic performance.

The DeLong test relies on several key assumptions:

• Asymptotic Normality: The test is valid for large samples; performance may be unreliable with very small case numbers.
• Identical Cases: All models must be evaluated on the exact same set of subjects. Missing data for one model violates the correlation structure.
• Binary Outcome: The standard formulation requires a binary ground truth (diseased vs. non-diseased). Extensions exist for ordinal outcomes.
• Independent Observations: The test assumes cases are independent; it does not natively handle clustered data (e.g., multiple lesions per patient) without modification.

The DeLong test is widely available in statistical and machine learning libraries:

• R: The pROC package provides the roc.test() function with method="delong".
• Python: The compare_auc_delong_xu function in the mlxtend library implements the test. The scikit-learn ecosystem does not have a native implementation, but custom code using the U-statistic formulation is straightforward.
• MedCalc: A dedicated medical statistics software that offers a graphical interface for DeLong comparisons.
• STATA: The roccomp command performs the test.