Reader Study

The defining characteristic of a reader study is the crossed design where multiple readers interpret the same set of cases. This factorial structure allows for the simultaneous estimation of reader variability and case variability. Unlike a simple cohort study, an MRMC design enables statistical inference that generalizes to both the population of readers and the population of cases, preventing the results from being biased by a single expert or an unrepresentative sample of images.

Reader studies almost universally employ a fully crossed factorial design. Every reader interprets every case under each modality (e.g., unaided vs. AI-assisted). This within-subject comparison dramatically increases statistical power by eliminating between-reader confounding. A mandatory washout period—often 4-8 weeks—is inserted between reading sessions to mitigate recall bias, ensuring that a reader's memory of a specific case does not artificially inflate the agreement between the two modalities.

The validity of a reader study hinges on the ground truth. Cases must be adjudicated by an independent panel using a reference standard that is superior to the test under evaluation. Common truthing methods include:

• Histopathological confirmation via biopsy
• Composite consensus panel of subspecialty experts
• Longitudinal clinical follow-up to confirm benignity An imperfect reference standard introduces incorporation bias, which can invalidate the study's sensitivity and specificity estimates.

To achieve adequate statistical power without requiring tens of thousands of cases, reader studies use enriched sampling. The prevalence of disease in the study set is artificially inflated—often to 30-50%—compared to the real-world screening population. This enrichment must be meticulously documented, as it directly impacts positive predictive value (PPV) and negative predictive value (NPV). Sensitivity and specificity, however, remain theoretically independent of prevalence and are the primary endpoints.

The study protocol must explicitly define the interaction model between the reader and the AI:

• Concurrent (Second Reader): The AI's marks and scores are displayed to the radiologist in real-time during their initial interpretation.
• Sequential (CADt): The reader first interprets the case unaided and records their findings. The AI output is then revealed, and the reader can amend their final report. The choice of mode fundamentally changes the clinical workflow being evaluated and the type of human-AI interaction being measured.

Reader studies are often structured as non-inferiority trials when AI aims to replace a human reader (e.g., triaging negative cases). The study must pre-specify a non-inferiority margin (delta)—the clinically acceptable degree of performance loss. If the confidence interval for the difference in AUC between AI and the human standard lies entirely above this margin, non-inferiority is established. For AI-as-assistance, a superiority design is used to prove the combination is statistically better than the human alone.