Inter-Rater Variability (Cohen's Kappa)

A statistical coefficient that measures inter-rater reliability for categorical items, correcting for the probability of agreement occurring by random chance. Unlike simple percent agreement, Cohen's Kappa isolates the true concordance between annotators. The formula is κ = (p₀ - pₑ) / (1 - pₑ), where p₀ is the observed proportional agreement and pₑ is the hypothetical probability of chance agreement based on each rater's observed category frequencies.

Landis and Koch (1977) provide the standard qualitative interpretation framework for Kappa values:

• < 0.00: Poor agreement (worse than chance)
• 0.00–0.20: Slight agreement
• 0.21–0.40: Fair agreement
• 0.41–0.60: Moderate agreement
• 0.61–0.80: Substantial agreement
• 0.81–1.00: Almost perfect agreement

In medical image segmentation, a κ ≥ 0.80 is typically the target for ground truth validation.

When segmentation disagreements are not equally severe—for instance, confusing adjacent anatomical structures versus missing a tumor entirely—Weighted Cohen's Kappa applies a penalty matrix. Linear weights penalize disagreements proportionally to the ordinal distance between categories, while quadratic weights penalize by the square of the distance. This is critical for Organ-at-Risk (OAR) segmentation where certain boundary errors have higher clinical consequence than others.

An extension of Cohen's Kappa designed for scenarios with three or more annotators evaluating the same set of images. Fleiss' Kappa calculates the degree of agreement in classification over and above what would be expected by chance. This is essential for large-scale annotation projects where multiple radiologists contribute to a single ground truth dataset, ensuring statistical validity of the reference standard across an entire labeling team.

Two well-known paradoxes can distort Kappa interpretation:

• Prevalence Effect: When the true class distribution is highly imbalanced (e.g., 95% background, 5% lesion), Kappa can be misleadingly low despite high observed agreement.
• Bias Effect: When raters systematically differ in their frequency of labeling a category, Kappa can be inflated.

Mitigation strategies include reporting sensitivity and specificity alongside Kappa and using Gwet's AC1 coefficient as a more stable alternative.

While Cohen's Kappa measures categorical agreement between raters, the Dice Score (F1 Score) measures spatial overlap between segmentation masks. In practice, Kappa is used during the annotation quality assurance phase to validate the ground truth, while Dice is used during model evaluation to compare predictions against that validated ground truth. A low Kappa among annotators sets a hard ceiling on achievable Dice scores, making it a foundational upstream metric.