Inter-Annotator Agreement (IAA)

A chance-corrected measure of agreement between exactly two annotators for categorical items.

• Mechanism: Compares observed agreement (Po) to expected agreement (Pe) based on marginal distributions.
• Formula: κ = (Po - Pe) / (1 - Pe)
• Use Case: Ideal for a single reviewer verifying a senior pharmacovigilance officer's causality assessments.
• Interpretation: κ > 0.8 indicates almost perfect agreement; κ < 0.4 suggests poor agreement beyond chance.

A statistical generalization of Cohen's Kappa for assessing agreement among three or more fixed raters.

• Mechanism: Calculates the degree of consensus in assignment over multiple categories, correcting for random agreement.
• Constraint: Assumes all items are rated by the same set of annotators; not suitable for varying rater pools.
• Pharmacovigilance Application: Used when a committee of medical experts classifies the same set of ICSRs for seriousness criteria.

A highly versatile reliability coefficient applicable to any number of raters, incomplete data, and multiple metric levels (nominal, ordinal, interval, ratio).

• Mechanism: A generalized chance-corrected agreement measure based on observed and expected disagreement.
• Key Advantage: Handles missing data gracefully, making it robust for real-world annotation projects where not every annotator labels every document.
• Gold Standard: Preferred in computational linguistics for complex pharmacovigilance entity extraction tasks.

The simplest metric, calculated as the raw proportion of matching labels out of the total annotations.

• Formula: (Number of Agreements) / (Total Annotations) × 100
• Critical Flaw: Does not correct for random chance. High agreement can be misleading for imbalanced datasets.
• Example: Two annotators may achieve 90% agreement on a rare adverse event simply by labeling everything as 'negative', masking true disagreement.
• Recommendation: Use only for quick exploratory analysis, never as a final reliability benchmark.

An extension of Cohen's Kappa that accounts for the degree of disagreement between ordered categories.

• Mechanism: Applies a penalty matrix where disagreements between distant ordinal classes (e.g., 'mild' vs. 'severe') are penalized more heavily than adjacent ones.
• Weighting Schemes: Linear weights penalize proportionally; quadratic weights penalize by the square of the distance.
• Use Case: Evaluating agreement on the severity grading of adverse events or the causality assessment scale.

A parametric statistic measuring the consistency or absolute agreement of quantitative measurements made by different observers.

• Mechanism: Derived from ANOVA models, comparing the variance of interest to total variance.
• Forms: ICC(1,1) for single measures; ICC(1,k) for average of k raters. 'Consistency' vs. 'Absolute Agreement' definitions differ.
• Application: Assessing reliability of continuous scores, such as the extraction of a numerical lab value from a clinical narrative.

A statistical coefficient measuring inter-rater reliability for two annotators classifying items into mutually exclusive categories. Unlike simple percent agreement, it accounts for agreement occurring by chance.

• Range: -1 (perfect disagreement) to +1 (perfect agreement)
• Interpretation: κ > 0.8 indicates strong agreement; κ < 0.4 suggests poor agreement
• Formula: κ = (p₀ - pₑ) / (1 - pₑ), where p₀ is observed agreement and pₑ is expected chance agreement
• Limitation: Sensitive to prevalence; high agreement on a rare category can paradoxically yield low kappa

A generalization of Cohen's Kappa for measuring agreement among three or more annotators assigning categorical ratings to a fixed number of items. Essential for large-scale pharmacovigilance annotation projects with multiple reviewers.

• Fixed marginal: Assumes annotators are not uniquely identified; treats all raters as interchangeable
• Weighting schemes: Can incorporate ordinal weights to penalize severe disagreements more than minor ones
• Use case: Calculating overall IAA when 5+ clinical reviewers label adverse event mentions across thousands of documents

A highly versatile reliability coefficient that handles any number of annotators, any metric level of measurement (nominal, ordinal, interval, ratio), and missing data. Considered the gold standard for content analysis reliability.

• Robustness: Accommodates incomplete annotation matrices where not every annotator labels every item
• Distance functions: Custom disagreement weights can be defined for domain-specific severity gradations
• Bootstrapping: Confidence intervals can be computed via resampling to assess estimate stability

The simplest IAA metric: the proportion of annotation instances where two or more annotators assigned identical labels. While intuitive, it is not recommended as a standalone measure because it fails to correct for chance agreement.

• Calculation: (Number of agreements / Total annotations) × 100
• Inflated estimates: High base rates of a common category artificially boost agreement scores
• Supplementary use: Often reported alongside kappa or alpha for transparency, but never relied upon exclusively for quality assurance

A formal document defining the label taxonomy, inclusion/exclusion criteria, and edge case resolution protocols that annotators must follow. The single most critical factor influencing IAA scores.

• Iterative refinement: Guidelines should be updated based on disagreement analysis during pilot annotation rounds
• Adjudication rules: Must specify how tie-breaking decisions are made when annotators disagree
• Pharmacovigilance specifics: Define precisely what constitutes an adverse event mention, seriousness criteria boundaries, and causality language interpretation

The process of resolving annotation disagreements to produce a final gold standard label. Typically performed by a senior domain expert (e.g., a drug safety physician) who reviews conflicting annotations and determines the definitive classification.

• Adjudicator qualifications: Must possess deeper domain expertise than the original annotators
• Blind adjudication: Adjudicator should ideally be blinded to which annotator produced which label to avoid bias
• Disagreement taxonomy: Track whether disagreements stem from guideline ambiguity, annotator error, or genuinely borderline cases to drive guideline improvement