ROC-AUC

ROC-AUC evaluates discriminative power across all possible classification thresholds, not just a single operating point. This is critical in medical imaging where the optimal sensitivity/specificity trade-off depends on clinical context.

• Plots True Positive Rate (Sensitivity) against False Positive Rate (1 - Specificity)
• Each point on the curve represents a different decision threshold
• An AUC of 1.0 indicates perfect separation; 0.5 indicates random guessing
• Enables comparison of models without pre-committing to a specific sensitivity/specificity balance

Unlike Positive Predictive Value (PPV) and Negative Predictive Value (NPV), ROC-AUC is mathematically independent of disease prevalence in the test population. This makes it the preferred metric for comparing diagnostic algorithms across different clinical settings.

• PPV and NPV shift dramatically with prevalence changes
• ROC-AUC remains stable whether disease prevalence is 1% or 50%
• Essential for external validation studies where case mix differs from training data
• Allows direct comparison of model performance across institutions with different patient demographics

When comparing two diagnostic models evaluated on the same set of cases, the DeLong test provides a non-parametric method to determine if the difference between their AUC values is statistically significant.

• Accounts for the correlated nature of ROC curves from the same subjects
• Produces a p-value for the null hypothesis that both models have equal AUC
• Required in reader studies comparing AI-assisted vs. unassisted interpretation
• Critical for demonstrating that a new model is non-inferior or superior to an established standard

AUC values provide a standardized scale for assessing discriminative ability, with specific ranges carrying clinical implications for regulatory review.

• 0.90–1.00: Excellent discrimination; typical for well-defined radiological tasks
• 0.80–0.90: Good discrimination; acceptable for screening applications
• 0.70–0.80: Fair discrimination; may require complementary biomarkers
• 0.50–0.70: Poor discrimination; insufficient for standalone clinical use
• Values below 0.50 indicate the model is systematically inverting predictions

ROC-AUC has known limitations that must be addressed in clinical validation study design. It summarizes global performance but obscures region-specific behavior critical to medical decision-making.

• Does not reflect clinical utility: A high AUC does not guarantee improved patient outcomes
• Insensitive to calibration: A model with excellent discrimination may still produce poorly calibrated probability estimates
• Weights all thresholds equally: In practice, only a narrow range of high-specificity thresholds is clinically relevant for screening
• Partial AUC (pAUC) is often preferred, focusing on the clinically relevant high-specificity region
• Supplement with Decision Curve Analysis to assess net benefit

ROC-AUC serves as a primary endpoint in many Software as a Medical Device (SaMD) clearance pathways. The FDA expects rigorous AUC reporting with confidence intervals and pre-specified performance goals.

• Must be accompanied by 95% confidence intervals derived from the DeLong method or bootstrap resampling
• Pre-specified primary analysis in the Statistical Analysis Plan (SAP) is mandatory
• Reader-averaged AUC is standard for MRMC studies comparing radiologists with and without AI assistance
• Sensitivity and specificity at a pre-defined operating point must also be reported as secondary endpoints