Cohen's Kappa

Cohen's Kappa's core innovation is correcting for agreement by chance. Simple percent agreement can be misleadingly high if categories are imbalanced. Kappa calculates the observed agreement (Po) and subtracts the probability of expected agreement (Pe) due to random chance, then normalizes the result: κ = (Po - Pe) / (1 - Pe). A κ of 0 indicates agreement no better than chance, while 1 indicates perfect agreement.

The value of Kappa is interpreted using established benchmarks. The widely cited scale by Landis and Koch (1977) provides qualitative labels:

• < 0.00: Poor agreement
• 0.00 – 0.20: Slight agreement
• 0.21 – 0.40: Fair agreement
• 0.41 – 0.60: Moderate agreement
• 0.61 – 0.80: Substantial agreement
• 0.81 – 1.00: Almost perfect agreement This scale helps contextualize Kappa's numerical output for practical decision-making in data annotation quality control.

Cohen's Kappa is designed for nominal or ordinal categorical data where raters assign items to mutually exclusive categories. Common use cases in ML evaluation include:

• Sentiment labeling (Positive/Neutral/Negative)
• Topic classification for retrieved documents
• Relevance judgment (Relevant/Irrelevant) for IR benchmarks
• Intent classification in dialogue systems It is not suitable for continuous or interval-scaled measurements, where metrics like the Intraclass Correlation Coefficient (ICC) are more appropriate.

Kappa is calculated from a confusion matrix (contingency table) summarizing the cross-tabulation of two raters' judgments. For a binary case (Yes/No), the matrix has four cells: agreements (Yes/Yes, No/No) and disagreements (Yes/No, No/Yes). Observed Agreement (Po): Sum of the diagonal cells divided by total items. Expected Agreement (Pe): For each category, multiply the raters' marginal proportions and sum these products. This represents the probability of random agreement given each rater's category distribution.

While powerful, Kappa has known limitations that engineers must account for:

• Prevalence Paradox: Kappa values can be artificially low when one category is very prevalent (e.g., 95% 'Not Relevant' documents), even if raters agree highly. Metrics like Prevalence-Adjusted Bias-Adjusted Kappa (PABAK) can address this.
• Bias Paradox: Disagreements are treated symmetrically, but systematic bias between raters affects interpretation.
• Number of Categories: Kappa tends to be lower with more categories, as chance agreement decreases. Weighted Kappa is used for ordinal data where some disagreements (e.g., 'Positive' vs. 'Neutral') are less severe than others ('Positive' vs. 'Negative').

Cohen's Kappa is part of a family of inter-rater reliability statistics:

• Fleiss' Kappa: Generalizes Cohen's Kappa to more than two raters.
• Krippendorff's Alpha: A more robust metric that handles missing data, any number of raters, and different measurement levels (nominal, ordinal, interval, ratio). It is often preferred for complex annotation tasks.
• Intraclass Correlation Coefficient (ICC): Used for continuous data, measuring agreement based on variance components from ANOVA. Choosing the correct metric depends on the data type, number of raters, and the need for tolerance to missing values.

Inter-Annotator Agreement is the broader statistical concept measuring the consensus or reliability among human labelers when categorizing data. It is the foundational requirement for creating high-quality ground-truth datasets used to train and evaluate machine learning models, including RAG retrievers and generators.

• Purpose: Ensures that evaluation benchmarks are based on consistent, reproducible human judgments.
• Methods: Encompasses several specific statistics, including Cohen's Kappa, Fleiss' Kappa (for multiple raters), and Krippendorff's Alpha (for handling missing data).
• RAG Application: Used to validate the labeling of query-document relevance pairs, the correctness of generated answers, and the identification of hallucinations.

Fleiss' Kappa is a statistical measure of inter-rater reliability for categorical items when there are more than two raters. It generalizes Scott's Pi statistic and, like Cohen's Kappa, corrects for agreement expected by chance.

• Key Difference from Cohen's Kappa: Designed for multiple raters (e.g., 3+ annotators), whereas Cohen's Kappa is strictly for two.
• Calculation: Based on the overall proportion of pairwise agreement among all raters.
• Use Case: Ideal for large-scale data labeling projects where multiple annotators label the same items to establish a robust ground truth for RAG evaluation benchmarks.

Krippendorff's Alpha is a versatile reliability coefficient for measuring agreement among multiple raters, applicable to any level of measurement (nominal, ordinal, interval, ratio), any number of raters, and robust to missing data.

• Key Advantage: Can handle incomplete datasets where not every rater evaluates every item, a common scenario in crowd-sourced annotation.
• Flexibility: Works with different data types and distance metrics, making it suitable for complex annotation tasks beyond simple categories.
• RAG Application: Useful for evaluating agreement on ordinal scales (e.g., ranking document relevance on a 1-5 scale) or when annotation tasks are distributed among many raters.

Precision and Recall are the fundamental binary classification and information retrieval metrics that form the basis for many agreement statistics. Understanding their trade-off is crucial for interpreting what Cohen's Kappa measures.

• Precision: The proportion of items a rater labeled as positive that are actually positive (according to a ground truth or another rater). Measures exactness.
• Recall: The proportion of all actually positive items that the rater successfully labeled. Measures completeness.
• Connection to Kappa: Cohen's Kappa synthesizes the observed agreement (a function of true positives, etc.) and contrasts it with chance agreement, providing a single score that accounts for the base rates of the categories.

The F1 Score is the harmonic mean of precision and recall, providing a single metric that balances both concerns. While used for system evaluation, it conceptually relates to the agreement between a system's output and a ground truth label.

• Formula: F1 = 2 * (Precision * Recall) / (Precision + Recall).
• Comparison with Kappa: Both produce a single score between 0 and 1. However, F1 measures accuracy against a ground truth, while Kappa measures agreement between two labelers, correcting for chance. F1 does not account for the possibility of agreement by random guessing.

A Confusion Matrix (or error matrix) is a specific table layout that visualizes the performance of a classification algorithm or, in the context of IAA, the agreement between two raters. It is the foundational data used to calculate Cohen's Kappa.

• Structure: Rows represent the labels from Rater A, columns represent labels from Rater B. Cells show counts of items falling into each agreement/disagreement category.
• Direct Input for Kappa: The counts in the matrix's diagonal (agreements) and off-diagonals (disagreements) are used directly in the Kappa formula.
• Utility: Provides immediate visual insight into the type of disagreements (e.g., which specific categories are most confused), which the single Kappa statistic does not reveal.