Bland-Altman Plot

The core mechanism plots the difference between two paired measurements (Method A - Method B) on the Y-axis against the mean of the two measurements ([A + B]/2) on the X-axis. This structure avoids the spurious correlation found in simple scatter plots and directly visualizes the magnitude of disagreement across the measurement range. The mean serves as the best available estimate of the true value when neither method is a perfect gold standard.

Bias is the systematic error between methods, calculated as the mean of all paired differences (d̄). It quantifies whether one method consistently overestimates or underestimates relative to the other. A bias close to zero indicates no systematic discrepancy. The bias line is plotted as a horizontal reference on the chart, allowing immediate visual assessment of directional error.

The 95% Limits of Agreement define the interval within which 95% of future differences between the two methods are expected to fall. Calculated as d̄ ± 1.96 × SD_diff, where SD_diff is the standard deviation of the differences. These bounds provide a clinically actionable range—if the LoA are narrower than a pre-defined clinical tolerance, the methods can be used interchangeably.

A critical visual check for heteroscedasticity—whether the differences increase or decrease as the magnitude of measurement changes. A funnel-shaped pattern or a significant slope in the regression of differences on means indicates proportional bias. This violates the assumption of constant variance and may require log-transformation of data or the use of percentage differences rather than absolute differences.

The Bland-Altman framework can be extended to evaluate within-method repeatability by plotting repeated measurements from the same method against each other. If the limits of agreement for a single method's repeated measures are wide relative to the between-method LoA, the measurement error of the individual methods dominates, making meaningful agreement assessment impossible.

Statistical agreement does not imply clinical agreement. The LoA must be compared against a pre-specified clinical margin of acceptability defined by domain experts. For example, in blood pressure monitoring, a difference of ±5 mmHg may be clinically irrelevant. If the 95% LoA fall entirely within this margin, the new method is validated for clinical interchangeability.

The Bland-Altman plot is the standard method for assessing agreement when a diagnostic AI outputs a continuous variable—such as tumor volume, ejection fraction, or bone density—rather than a binary classification. It plots the difference between the AI and reference measurements (y-axis) against the mean of the two measurements (x-axis) for each case. This visual approach immediately reveals whether the AI systematically over- or underestimates values across the measurement range, a critical insight that a simple correlation coefficient would mask.

The plot provides three essential summary statistics:

• Bias (Mean Difference): The average of all differences, indicating systematic error. A bias close to zero suggests no fixed error.
• Limits of Agreement (LoA): Calculated as Bias ± 1.96 × Standard Deviation of the differences. This interval is expected to contain 95% of future measurement discrepancies.
• Confidence Intervals: Precision estimates for both the bias and LoA, essential for determining if the observed disagreement is clinically acceptable. For a diagnostic AI to be considered interchangeable with a reference standard, the LoA must fall within a pre-defined clinical tolerance margin.

A key diagnostic feature of the plot is its ability to expose proportional bias—a scenario where the disagreement between methods changes as a function of the measurement magnitude. This is visually identified as a funnel shape or a non-zero slope in the scatter of differences. For example, an AI for cardiac volumetry might agree well with MRI on normal-sized hearts but increasingly underestimate volumes in enlarged hearts. A formal test for this is to regress the differences against the means; a statistically significant slope confirms proportional bias and invalidates a single summary bias statistic.

A high Pearson correlation coefficient (r) is often mistakenly used to claim agreement, but it measures only the strength of a linear relationship, not actual measurement concordance. Two methods can have a perfect correlation of r=0.99 yet exhibit a systematic bias of 10 units. The Bland-Altman plot is the correct, peer-reviewed alternative for method comparison studies. Regulatory bodies and clinical journals now expect this analysis in validation reports for quantitative imaging biomarkers, making it a mandatory component of a rigorous AI validation framework.

The standard Bland-Altman analysis assumes a single pair of measurements per subject. In AI validation, it's common to have repeated measurements (e.g., multiple radiologists, test-retest scans). A modified approach is required to avoid underestimating the true variance. The mixed-effects model extension calculates variance components for subjects, methods, and random error separately. This allows for the calculation of LoA that correctly account for the study's hierarchical design, providing a more honest assessment of the AI's agreement with the reference standard in real-world, noisy clinical environments.

Before plotting, a study must define a priori clinical acceptance criteria—the maximum allowable difference between the AI and the reference standard that would not affect clinical decision-making. This margin is used to interpret the LoA. If the upper and lower LoA fall entirely within this pre-defined margin, agreement is confirmed. The required sample size for the study is calculated based on the desired precision for the LoA's confidence interval, ensuring the study is adequately powered to make a definitive claim about the AI's measurement interchangeability.