Inter-Annotator Agreement (IAA)

A robust metric for measuring agreement between two raters on categorical items, explicitly correcting for the probability of chance agreement. Unlike raw percent agreement, Kappa penalizes random consensus.

• Formula: κ = (p₀ - pₑ) / (1 - pₑ), where p₀ is observed agreement and pₑ is expected chance agreement
• Interpretation: κ > 0.80 indicates near-perfect agreement; κ < 0.40 signals poor reliability
• Use case: Ideal for paired review of binary clinical classifications like 'condition present' vs. 'absent'

A generalization of Cohen's Kappa that assesses the reliability of agreement among three or more raters assigning categorical ratings to a fixed number of items. It is the standard for multi-reviewer annotation panels.

• Fixed-marginal assumption: Treats raters as interchangeable, not uniquely identified
• Benchmarking: Essential for validating annotation guidelines across a team of clinical coders
• Limitation: Sensitive to trait prevalence; high agreement on rare categories can paradoxically lower Kappa

A highly versatile agreement coefficient that supports any number of raters, any metric level (nominal, ordinal, interval, ratio), and handles missing data gracefully. It is the gold standard for complex annotation study designs.

• Distance-based: Can weight disagreements by severity, not just binary match/mismatch
• Robustness: Bootstrapped confidence intervals quantify estimate uncertainty
• Adoption: Increasingly preferred in NLP research for its mathematical generality over legacy metrics

The simplest IAA metric: the proportion of items on which raters assign identical labels. While intuitive, it is deceptively optimistic because it ignores chance agreement.

• Calculation: (Number of agreements / Total items) × 100
• Critical flaw: Two untrained raters guessing randomly on a binary task will still achieve ~50% agreement
• Appropriate use: Only as a preliminary sanity check before applying chance-corrected coefficients like Kappa or Alpha

The preferred IAA metric for continuous or ordinal data, quantifying both the degree of correlation and agreement between raters. ICC models differ based on whether raters are considered a random or fixed effect.

• ICC(2,1): Two-way random-effects model for generalizing to a population of raters
• ICC(3,1): Two-way mixed-effects model for a specific, fixed panel of raters
• Clinical application: Measuring agreement on ejection fraction percentages or lesion size measurements

A statistical coefficient measuring agreement between two raters for categorical items, correcting for agreement occurring by chance. It is the standard metric for simple adjudication tasks.

• Formula: κ = (p_o - p_e) / (1 - p_e), where p_o is observed agreement and p_e is expected chance agreement.
• Interpretation: Values range from -1 to 1. A score > 0.80 indicates strong agreement, while < 0.40 suggests poor reliability.
• Limitation: Not suitable for tasks involving more than two annotators.

A generalization of Cohen's Kappa for measuring agreement among three or more raters assigning categorical ratings to a fixed number of items. It is essential for multi-reviewer consensus studies.

• Use Case: Evaluating agreement across a panel of clinicians labeling the same set of radiology reports.
• Assumption: Raters are considered interchangeable and not uniquely identified.
• Advantage: Handles incomplete data where not every rater evaluates every subject.

A highly versatile reliability coefficient applicable to any number of raters, any metric level of measurement (nominal, ordinal, interval, ratio), and incomplete data sets.

• Robustness: Handles missing data gracefully without requiring a fully crossed design.
• Custom Distance Functions: Allows defining the severity of disagreement, making it ideal for clinical tasks where a near-miss is better than a complete mismatch.
• Gold Standard: Considered the most statistically rigorous IAA metric for complex annotation schemas.

A parametric statistic used to assess absolute agreement or consistency for continuous, quantitative data. It is the preferred metric for evaluating agreement on measurements like tumor size or ejection fraction.

• Model Types: Two-way mixed-effects (for fixed raters) vs. two-way random-effects (for generalizable raters).
• Form: ICC(3,1) measures consistency of a single rater, while ICC(3,k) measures the reliability of the mean of k raters.
• Threshold: An ICC > 0.90 is typically required for high-stakes clinical measurement tasks.

A formal, written document defining the explicit rules and edge-case logic that human annotators must follow to ensure consistent labeling. It is the single most critical artifact for achieving high IAA.

• Components: Includes entity definitions, span boundary rules, negation handling, and detailed examples of ambiguous cases.
• Iteration: Must be version-controlled and updated based on discrepancy analysis during the adjudication workflow.
• Impact: Ambiguous guidelines are the primary root cause of low inter-annotator agreement and reviewer drift.

A structured adjudication workflow where multiple annotators discuss and resolve disagreements to produce a single, authoritative ground truth label.

• Process: Begins with independent annotation, measures IAA, then escalates discrepancies to a senior reviewer or a committee vote.
• Outcome: Generates a golden dataset used for model training and evaluation.
• Benefit: Resolves genuine ambiguity in clinical text, transforming subjective interpretation into a standardized reference standard.