Cohen's Kappa

Fleiss' Kappa is a statistical measure that generalizes Cohen's Kappa to assess the reliability of agreement among three or more raters or models when assigning categorical ratings. It calculates the degree of agreement beyond what is expected by chance for a fixed number of raters across multiple items.

• Key Difference: While Cohen's Kappa is for two raters, Fleiss' Kappa handles multiple raters.
• Use Case: Ideal for evaluating consensus in crowdsourced labeling tasks or the outputs of a multi-agent ensemble where more than two entities provide judgments.

Truth inference is the broader process of aggregating multiple, potentially noisy or conflicting labels or predictions from different sources—such as crowd workers, sensors, or machine learning models—to estimate a single, reliable 'ground truth' label.

• Relation to Kappa: Cohen's Kappa measures the agreement between two such sources, which is a foundational input for many truth inference algorithms.
• Algorithms: Common methods include Dawid-Skene, Majority Voting, and Weighted Consensus, which often use agreement statistics to weight source reliability.

Weighted consensus is an aggregation technique where the contributions of individual models, agents, or raters are combined based on assigned confidence or reliability weights to form a final output.

• Mechanism: Weights can be derived from historical accuracy, self-reported confidence scores, or, crucially, from inter-rater agreement metrics like Cohen's Kappa.
• Application: In agentic systems, a planner might weight the suggestions of different sub-agents based on their past agreement with a verifier, using a Kappa-like score to dynamically adjust influence.

Inter-rater reliability is the general concept of measuring the degree of agreement or consistency among two or more independent raters, judges, or evaluation systems. Cohen's Kappa is one specific statistic within this family.

• Other IRR Metrics: Includes Percent Agreement, Intraclass Correlation Coefficient (ICC) for continuous data, and Krippendorff's Alpha, which is more robust for small samples and handles missing data.
• Engineering Role: A core concern in building evaluation frameworks for AI systems, ensuring that performance benchmarks and quality checks are consistently applied.

A confusion matrix is a specific table layout used to visualize the performance of a classification algorithm, showing counts for true positives, false positives, true negatives, and false negatives. It is the primary input for calculating Cohen's Kappa.

• Calculation Link: Cohen's Kappa uses the observed agreement (the sum of the diagonal) and the expected agreement (calculated from the row and column totals) from the confusion matrix.
• Foundation: Understanding the confusion matrix is essential for interpreting not just Kappa, but also metrics like precision, recall, and F1-score.

Bayesian Model Averaging is a rigorous probabilistic framework for combining predictions from multiple competing models by weighting them according to their posterior model probability given the observed data. It accounts for model uncertainty.

• Contrast with Simple Agreement: While Cohen's Kappa measures observed agreement, BMA provides a principled way to aggregate different models' outputs based on how well they explain the data, inherently correcting for chance in a Bayesian sense.
• Use in Self-Consistency: Can be used to aggregate the diverse reasoning paths or 'thoughts' generated by a single LLM using techniques like Tree-of-Thoughts, where each path is treated as a model.