Repeatability

Repeatability requires unchanged conditions across successive measurements. This means the same operator, the same measuring system, the same operating conditions, the same location, and repetition over a short period of time. In diagnostic AI, this translates to running the same model inference on the exact same input data multiple times without any alteration to hardware, software environment, or preprocessing pipeline. Any variation in output under these conditions is attributable solely to the measurement system's inherent noise.

Repeatability is often confused with reproducibility, but they address different sources of variation:

• Repeatability: Same operator, same equipment, same conditions, short interval. Measures inherent instrument precision.
• Reproducibility: Different operators, different equipment, different locations. Measures real-world robustness. In medical imaging AI, repeatability assesses whether the model produces identical outputs on the same scan twice in a row. Reproducibility assesses whether a different hospital running the model on a different scanner produces clinically equivalent results.

Repeatability is expressed statistically through metrics of dispersion:

• Coefficient of Variation (CV): The ratio of the standard deviation to the mean, often expressed as a percentage. A CV < 5% is a common target for quantitative imaging biomarkers.
• Repeatability Coefficient (RC): Defined as 1.96 × √2 × within-subject standard deviation, representing the maximum difference likely to occur between two repeated measurements with 95% confidence.
• Intraclass Correlation Coefficient (ICC): A reliability index ranging from 0 to 1, where values > 0.90 indicate excellent repeatability for continuous measurements.

Even deterministic AI models can exhibit non-repeatability due to subtle environmental factors:

• Floating-point non-determinism: GPU parallel reductions and non-associative floating-point arithmetic can produce slightly different results across runs, especially on different hardware architectures.
• Random seed leakage: Uncontrolled stochastic elements in preprocessing, such as random cropping or augmentation applied during inference.
• Software stack variability: Differences in deep learning framework versions, CUDA versions, or driver-level optimizations.
• Hardware thermal throttling: Performance degradation under sustained load can alter execution paths in just-in-time compiled kernels.

Poor repeatability undermines clinical confidence in quantitative imaging biomarkers. For example, if an AI system measures tumor volume with a repeatability coefficient of ±15%, a measured volume change of 10% between two scans cannot be distinguished from measurement noise. This directly impacts:

• Treatment response assessment: RECIST criteria rely on detecting meaningful changes over time.
• Longitudinal tracking: Unreliable measurements obscure true disease progression or regression.
• Regulatory acceptance: FDA and EMA require demonstrated repeatability as part of analytical validation for SaMD submissions.

Engineering practices to guarantee repeatable AI inference:

• Containerization: Docker images with pinned dependency versions eliminate software stack variability.
• Deterministic mode flags: Setting torch.use_deterministic_algorithms(True) and CUBLAS_WORKSPACE_CONFIG in PyTorch enforces deterministic CUDA operations.
• Fixed random seeds: Explicitly setting seeds for all random number generators in the inference pipeline.
• Bitwise output validation: Implementing checksum verification on model outputs to detect any deviation across deployments.
• Continuous monitoring: Automated drift detection comparing current outputs against a golden reference dataset.

The closeness of agreement between results of measurements of the same measurand carried out under changed conditions. While repeatability requires identical conditions (same operator, same lab, short interval), reproducibility intentionally varies factors like different operators, different laboratories, or different equipment to assess real-world robustness.

• Key distinction: Repeatability = same conditions; Reproducibility = different conditions
• Often assessed through multi-site external validation studies
• Critical for demonstrating that a diagnostic AI model works across diverse clinical environments

The ability of a diagnostic test to accurately and reliably measure the analyte or biomarker of interest under specified laboratory conditions. Repeatability is a core component of analytical validity, alongside accuracy, precision, and limits of detection.

• Establishes the foundational technical performance before clinical utility is assessed
• Required by FDA for Software as a Medical Device (SaMD) submissions
• Poor repeatability undermines all downstream clinical validation efforts

A graphical method for comparing two measurement techniques by plotting the difference between paired measurements against their mean. Widely used to visualize repeatability by assessing the limits of agreement between repeated measurements on the same subjects.

• Displays bias (mean difference) and 95% limits of agreement
• Reveals whether disagreement varies with measurement magnitude
• Preferred over correlation coefficients for method comparison studies

The ratio of the standard deviation to the mean, expressed as a percentage, used to quantify repeatability in quantitative measurements. Lower CV values indicate superior repeatability.

• Formula: CV = (σ / μ) × 100%
• Commonly used in laboratory medicine for assay validation
• Enables comparison of precision across different measurement scales and units

A descriptive statistic that quantifies the degree of absolute agreement between multiple measurements of the same quantity. Unlike Pearson correlation, ICC penalizes systematic bias, making it the preferred metric for assessing repeatability and test-retest reliability.

• ICC(1,1): One-way random effects model for single measurements
• ICC(2,1): Two-way random effects model accounting for rater effects
• Values above 0.90 generally indicate excellent repeatability

The objective, verified diagnosis established by an independent reference standard against which a diagnostic test is evaluated. Repeatability measurements are only meaningful when compared against a stable ground truth that does not itself vary between repeated measurements.

• Must be established independently of the test under evaluation
• Common standards include histopathological confirmation or consensus expert panels
• Unstable ground truth can masquerade as poor test repeatability