Reader Study

The foundational statistical framework for evaluating diagnostic imaging devices. The study must include multiple readers (radiologists) interpreting multiple cases (patient exams) to ensure results generalize beyond a specific individual or image set.

• Readers: A representative sample of the target user population, varying in experience and subspecialty.
• Cases: An enriched but representative sample, including subtle lesions, normals, and confounding benign findings.
• Crossing: Every reader interprets every case in both unaided and AI-assisted modes, separated by a washout period (typically 4-6 weeks) to mitigate memory bias.

The case set must be statistically powered and clinically representative. A random screening cohort would require an impractically large sample size due to low cancer prevalence.

• Enrichment: Artificially increase disease prevalence (e.g., 25-50% malignant) to achieve statistical power with a feasible number of cases.
• Ground Truth: Establish definitive truth via histopathological biopsy for malignant cases and 2-year negative follow-up for benign/normal cases.
• Spectrum Bias: Avoid including only obvious cancers; the set must contain subtle lesions, architectural distortions, and microcalcifications that challenge both readers and AI.

The reading paradigm defines how AI assistance is integrated and how bias is controlled.

• Sequential Design: Readers first interpret the exam unaided, record findings, then immediately view AI output and may revise. This mimics clinical workflow but introduces contamination bias.
• Crossover Design: Readers interpret all cases unaided in one session and all cases aided in a separate session after a washout. This is the gold standard for measuring standalone AI impact.
• Concurrent Reading: AI marks are displayed simultaneously during initial interpretation. This is the most clinically realistic but makes isolating the AI's specific contribution statistically complex.

The study must pre-specify a primary performance endpoint that captures the clinical task.

• FROC (Free-Response Operating Characteristic): The gold standard for detection tasks. Plots lesion-level true positive rate against the average false positives per image. Evaluates localization accuracy, not just image-level classification.
• ROC (Receiver Operating Characteristic): Plots sensitivity vs. specificity at the case level. Useful for diagnosis/classification tasks but ignores localization.
• JAFROC (Jackknife Alternative FROC): A specific statistical method for analyzing MRMC-FROC data that accounts for both reader and case variability.

The study hypothesis must be defined with a clinically justified statistical margin, pre-registered with the FDA.

• Superiority: Prove AI-assisted reading is statistically significantly better than unaided reading. Requires a larger sample size.
• Non-Inferiority: Prove AI-assisted reading is not worse than unaided reading by a pre-specified delta margin (e.g., 5% sensitivity). Often used when the primary claim is improved specificity or reading time.
• Co-Primary Endpoints: Common in AI studies, e.g., demonstrating non-inferior sensitivity AND superior specificity simultaneously, requiring alpha adjustment for multiplicity.

Readers must represent the intended user population, and their interaction with the AI must be standardized.

• Eligibility: Define minimum board certification, mammography volume (e.g., >480 reads/year per MQSA), and fellowship training.
• Training Phase: Include a dedicated training set (not part of the analysis) to familiarize readers with the AI interface and output format before the evaluated sessions begin.
• Calibration: Measure baseline unaided performance to ensure the reader pool is not an outlier relative to published benchmarks, validating generalizability.

The gold-standard experimental design for evaluating diagnostic AI where multiple radiologists interpret the same set of patient cases under different modalities. This crossed factorial structure accounts for both reader variability and case variability, enabling generalization of results beyond the specific clinicians or images in the study. Statistical analysis typically uses Dorfman-Berbaum-Metz (DBM) or Obuchowski-Rockette (OR) methods to properly model the correlation structure.

The primary endpoint in most reader studies, measuring a reader's discriminative ability to distinguish between diseased and non-diseased cases across all decision thresholds. An AUC of 0.5 represents chance performance, while 1.0 indicates perfect separation. In AI-assisted reading studies, the key comparison is the difference in AUC between unaided interpretation and AI-assisted interpretation, with a statistically significant increase demonstrating clinical benefit.

A study structure where each reader interprets every case under both conditions (unaided and AI-assisted), but with a washout period between sessions to minimize memory bias. Cases are split into two balanced sets, and readers are randomized to interpret Set A unaided first, then Set B with AI after a 4-6 week interval. This design maximizes statistical power by using each reader as their own control, reducing the confounding effect of inter-reader variability.

Co-primary endpoints that decompose diagnostic accuracy into two clinically meaningful components:

• Sensitivity: The proportion of true cancers correctly recalled, measuring the system's ability to avoid missed diagnoses
• Specificity: The proportion of normal cases correctly cleared, measuring the system's ability to avoid unnecessary recalls In screening mammography, a 1-2% improvement in specificity can translate to thousands of avoided false-positive workups annually at a population level.

The statistical framework defining the study's hypothesis structure. A non-inferiority trial aims to demonstrate that AI-assisted reading is not meaningfully worse than standard double-reading, using a pre-specified non-inferiority margin (e.g., ΔAUC < 0.05). A superiority trial seeks to prove AI assistance statistically outperforms unaided reading. Sequential testing allows first establishing non-inferiority, then testing for superiority without multiplicity penalty.

The deliberate oversampling of malignant cases in a reader study to achieve adequate statistical power, since natural screening populations have a cancer prevalence of only 0.3-0.5%. A typical enriched study uses a 50% cancer prevalence, but this artificial ratio must be accounted for when extrapolating sensitivity and specificity to real-world screening settings. Failure to adjust for enrichment can produce spectrum bias and overestimate clinical utility.