Cohen's Kappa

A statistical measure for assessing the reliability of agreement between a fixed number of raters when classifying categorical items. Unlike Cohen's Kappa, which is restricted to two raters, Fleiss' Kappa generalizes the concept to three or more raters. It corrects for agreement occurring by chance and is widely used in multi-reader studies to validate the consistency of diagnostic annotations across a panel of radiologists.

A variant of Cohen's Kappa that allows for differential weighting of disagreements. Instead of treating all misclassifications equally, it assigns a weight matrix based on the severity of the discrepancy.

• Linear weights: Penalty proportional to the distance between categories.
• Quadratic weights: Penalty proportional to the square of the distance. This is critical in ordinal diagnostic scales (e.g., BI-RADS scores) where a disagreement between adjacent categories is less severe than between distant ones.

A descriptive statistic used when measurements are continuous rather than categorical. The ICC assesses the consistency of quantitative measurements made by different observers measuring the same quantity. It is the preferred metric for test-retest reliability of volumetric tumor measurements or ejection fraction calculations, where Cohen's Kappa would be inappropriate due to data type.

A highly flexible reliability coefficient that generalizes across any number of raters, any metric level of measurement (nominal, ordinal, interval, ratio), and handles missing data gracefully. It is mathematically robust for incomplete datasets where not every rater evaluates every case. Krippendorff's Alpha is often preferred in content analysis and large-scale clinical annotation projects with irregular rater assignments.

The simplest measure of inter-rater reliability, calculated as the raw proportion of cases on which raters agree. While intuitive, it is fundamentally flawed because it does not correct for chance agreement. In a dataset with high disease prevalence, raters can achieve high percent agreement simply by guessing the majority class. Cohen's Kappa explicitly addresses this limitation by subtracting the expected chance agreement from the numerator.

A recalibration of Cohen's Kappa that addresses two paradoxes: the high agreement, low kappa paradox caused by extreme prevalence, and the bias effect caused by raters using categories at different frequencies. PABAK adjusts the observed agreement to neutralize these distortions, providing a more interpretable reliability metric when the underlying condition is rare or raters have asymmetric diagnostic tendencies.