Analytical Validity

Precision quantifies the closeness of agreement between independent test results obtained under stipulated conditions. It is not a single metric but a family of estimates.

• Repeatability: Variation observed when the same operator uses the same instrument to measure the identical sample multiple times within a short interval. It defines the baseline 'noise floor' of the assay.
• Reproducibility: Variation observed when conditions change—different operators, different reagent lots, or different laboratories. This is critical for multi-site AI deployment.
• Statistical Expression: Typically reported as a Coefficient of Variation (CV) , where a CV < 10% is often a target for high-complexity assays.

Accuracy is the degree of closeness of the measured value to the true value of the analyte. It is decomposed into two components:

• Trueness: The systematic difference between the average of a large series of measurements and the reference standard. High trueness means low systematic bias.
• Bias Assessment: Evaluated using Bland-Altman analysis or by measuring certified reference materials (CRMs).
• Clinical Context: An AI model can be highly precise (low random error) but inaccurate (high systematic bias), consistently missing the true tumor volume by a fixed margin. Both must be validated independently.

These parameters define the operational boundaries of an assay at the low end of the measurement range.

• Limit of Blank (LoB): The highest apparent analyte concentration expected when blank samples (containing no analyte) are tested. It defines the noise floor.
• Limit of Detection (LoD): The lowest analyte concentration that can be reliably distinguished from the LoB. It answers: 'Is the substance present?'
• Limit of Quantitation (LoQ): The lowest concentration at which the analyte can not only be detected but also measured with acceptable precision and accuracy. For AI imaging tools, this translates to the minimum lesion size (e.g., 3mm) that can be segmented reliably.

Linearity is the ability of the assay to provide results directly proportional to the concentration of the analyte within a given analytical measurement range (AMR) .

• Validation Protocol: Serial dilutions of a high-concentration sample are tested. The observed values are plotted against expected values; a polynomial regression analysis confirms if the deviation from linearity is clinically insignificant.
• Signal Saturation: In AI imaging, non-linearity occurs when pixel intensity (Hounsfield Units) saturates, causing the model to fail on very dense or very radiolucent materials.
• Clinical Reportability: Results outside the validated AMR cannot be reported quantitatively and must be flagged as '> upper limit' or '< lower limit'.

Analytical specificity is the ability to measure the intended analyte unequivocally in the presence of potential interferents or cross-reactants.

• Interference Testing: Common endogenous interferents (hemoglobin, bilirubin, lipids) or exogenous substances (contrast agents, medications) are spiked into samples to check for bias > 10%.
• Cross-Reactivity: In AI pathology, this refers to the model's ability to distinguish the target cell morphology from visually similar but clinically distinct mimics (e.g., reactive atypia vs. malignancy).
• Robustness: A highly specific assay is robust to sample degradation, fixation artifacts, and scanning protocol variations without generating spurious signals.

Stability defines the analyte's resilience over time and under stress, while robustness tests the assay's resilience to deliberate small variations.

• Sample Stability: Evaluates analyte integrity under storage conditions (room temp, refrigerated, frozen) and freeze-thaw cycles. Critical for retrospective imaging studies using archived DICOM data.
• Reagent/Model Drift: Monitors AI model performance over time. A sudden drop in confidence scores without a corresponding change in input data indicates concept drift.
• Robustness Testing (Youden): A factorial design experiment that deliberately varies multiple factors (e.g., scanner vendor, slice thickness, kernel) simultaneously to identify failure modes before deployment.

The fundamental building blocks of diagnostic performance. Sensitivity measures the test's ability to correctly identify patients with the disease (true positive rate), while Specificity measures its ability to correctly identify those without the disease (true negative rate).

• A highly sensitive test is ideal for screening to rule out disease.
• A highly specific test is crucial for confirming a diagnosis to avoid false positives.
• These metrics are intrinsic to the test and independent of disease prevalence.

The Receiver Operating Characteristic curve plots the True Positive Rate against the False Positive Rate at every possible classification threshold. The Area Under the Curve (AUC) provides a single, threshold-independent metric of a model's discriminative power.

• An AUC of 1.0 indicates perfect discrimination.
• An AUC of 0.5 indicates performance no better than random chance.
• The DeLong Test is the standard non-parametric method for statistically comparing the AUCs of two correlated models.

The objective reference standard against which a new diagnostic test is measured. A robust ground truth is non-negotiable for analytical validity.

• Often established via histopathological biopsy, surgical confirmation, or a consensus panel of expert readers.
• A flawed or noisy ground truth will invalidate all downstream performance metrics.
• The process must be independent and blinded from the AI model's output to prevent incorporation bias.

Unlike sensitivity and specificity, Positive Predictive Value (PPV) and Negative Predictive Value (NPV) are heavily influenced by the prevalence of the disease in the tested population.

• PPV: Given a positive test, what is the probability the patient truly has the disease?
• NPV: Given a negative test, what is the probability the patient truly is disease-free?
• A test with high analytical validity can still have a low PPV if deployed in a very low-prevalence setting.

These two concepts define the precision of a measurement under different conditions, a core component of analytical validity.

• Repeatability: Agreement of results under identical conditions (same operator, same device, short time interval).
• Reproducibility: Agreement under changed conditions (different operators, different laboratories, different scanner models).
• High reproducibility is critical for a diagnostic AI to be deployed across diverse clinical sites and hardware vendors.

The foundational contingency table that visualizes the raw performance of a classification algorithm by tabulating four key counts:

• True Positives (TP): Correctly identified disease cases.
• True Negatives (TN): Correctly identified non-disease cases.
• False Positives (FP): Non-disease cases incorrectly flagged as positive (Type I error).
• False Negatives (FN): Disease cases missed by the test (Type II error).

All other performance metrics are derived from these four values.