Fleiss' Kappa

Fleiss' Kappa extends Cohen's Kappa, which measures agreement between two raters, to scenarios involving three or more raters. This makes it essential for evaluating ensembles, multi-agent systems, or crowdsourced labeling tasks where more than two sources provide judgments.

• Key Formula: κ = (P̄ - P̄_e) / (1 - P̄_e)
• P̄: The observed proportion of agreement across all raters and items.
• P̄_e: The expected proportion of agreement due to chance, calculated from the marginal distributions of categories across all raters.

The core value of Fleiss' Kappa is its correction for agreement expected by random chance. A simple percentage agreement can be misleadingly high if categories are imbalanced. Fleiss' Kappa provides a more rigorous score where:

• κ = 1: Perfect agreement beyond chance.
• κ = 0: Agreement equal to what is expected by chance.
• κ < 0: Agreement worse than chance (systematic disagreement). This correction is vital for assessing the genuine consensus in self-consistency mechanisms like ensemble voting or label aggregation.

Unlike some other agreement statistics, Fleiss' Kappa assumes a fixed set of raters evaluates all items (or a sample thereof). This structure is common in AI evaluation setups, such as:

• Having a panel of 5 expert models score 100 agent responses.
• Using 3 different truth inference algorithms to label a dataset. The metric does not accommodate scenarios where different items are rated by different, varying subsets of raters, which requires alternative measures like Krippendorff's Alpha.

Fleiss' Kappa is designed for nominal (categorical) data where ratings fall into distinct, unordered classes. Examples in AI systems include:

• Intent classification (e.g., 'purchase', 'query', 'complaint').
• Sentiment labels (e.g., 'positive', 'neutral', 'negative').
• Error type categorization in model outputs. It is not suitable for ordinal (ranked) or continuous data without modification. For ordinal ratings, weighted Kappa variants that penalize disagreements by degree of distance are used.

While interpretation depends on context, established benchmarks from Landis & Koch (1977) are commonly cited:

• < 0.00: Poor agreement.
• 0.00 – 0.20: Slight agreement.
• 0.21 – 0.40: Fair agreement.
• 0.41 – 0.60: Moderate agreement.
• 0.61 – 0.80: Substantial agreement.
• 0.81 – 1.00: Almost perfect agreement. In agentic systems, a high Fleiss' Kappa between multiple agent instances indicates reliable, deterministic behavior, which is critical for production-grade deployment.

Fleiss' Kappa is part of a family of agreement and consensus mechanisms. Key distinctions:

• vs. Majority Voting: Fleiss' Kappa measures the reliability of the agreement that voting relies upon. Low Kappa suggests voting may be unstable.
• vs. Truth Inference: Fleiss' Kappa can be used to assess the consistency of the noisy sources before applying a truth inference algorithm like Dawid-Skene.
• vs. Intraclass Correlation Coefficient (ICC): ICC is used for continuous or ordinal data, while Fleiss' Kappa is for categorical data. It is a foundational diagnostic tool before implementing more complex aggregation techniques like weighted consensus or Bayesian Model Averaging.

Cohen's Kappa is a statistical measure of inter-rater agreement for two raters on a categorical scale, correcting for the level of agreement expected by chance. It is the foundational metric from which Fleiss' Kappa generalizes to multiple raters.

• Key Difference: While Fleiss' Kappa handles a fixed number of raters (≥2), Cohen's Kappa is specifically designed for pairwise agreement.
• Calculation: It compares the observed agreement (pₒ) to the expected agreement by chance (pₑ) using the formula: κ = (pₒ - pₑ) / (1 - pₑ).
• Use Case: Commonly used to evaluate the reliability of two human annotators or to compare a single model's output against a gold-standard label.

Truth inference is the process of aggregating multiple, potentially noisy labels from different sources (e.g., crowd workers, weak models, or sensors) to estimate a single, reliable 'ground truth' label. It is the practical engineering problem that metrics like Fleiss' Kappa help evaluate.

• Core Challenge: Different sources have varying levels of expertise, bias, and reliability.
• Common Algorithms: Methods include Dawid-Skene, Majority Voting, and more sophisticated probabilistic graphical models that estimate both the true label and each source's confusion matrix.
• Application: Critical for creating high-quality training datasets, evaluating model ensembles, and aggregating outputs in decentralized or multi-agent systems.

Majority voting (or hard voting) is a fundamental consensus mechanism where the final categorical output is determined by selecting the option chosen by the majority of individual models or agents in an ensemble. It is a simple aggregation method whose reliability can be assessed using Fleiss' Kappa.

• Mechanism: Each model in an ensemble gets one vote; the class with the most votes wins.
• Relation to Kappa: A high Fleiss' Kappa score among ensemble members indicates strong agreement, suggesting that majority voting will produce a stable and reliable result.
• Limitation: Treats all models as equally reliable, which may not be optimal if some models are more accurate than others.

Ensemble averaging is a self-consistency technique for regression or probability outputs that combines predictions from multiple models by computing their arithmetic mean. While Fleiss' Kappa measures categorical agreement, ensemble averaging is used to produce a final, more stable and accurate continuous prediction.

• Purpose: Reduces variance and mitigates the impact of outliers from any single model.
• Contrast with Voting: Used for numerical outputs, whereas majority voting is for categorical labels.
• Foundation for Uncertainty: The variance across the ensemble's predictions can be used as a simple measure of predictive uncertainty.

Inter-rater reliability (IRR) is the broader field of study concerned with the degree of agreement among raters. Fleiss' Kappa is one specific statistic within this field, designed for categorical data with multiple raters.

• Other IRR Metrics: Includes Intraclass Correlation Coefficient (ICC) for continuous data, Krippendorff's Alpha (which handles missing data and is applicable to any level of measurement), and Percent Agreement (which does not correct for chance).
• Engineering Significance: High IRR is a prerequisite for creating trustworthy labeled datasets, which are the foundation for supervised learning. Low IRR signals that annotation guidelines are unclear or the task is inherently ambiguous.

Weighted consensus is an aggregation technique where the contributions of individual models or agents are combined based on assigned weights, typically reflecting their estimated confidence, historical accuracy, or reliability. It is a more sophisticated alternative to simple majority voting.

• Mechanism: The final output is a weighted sum or a weighted vote, where higher-performing models have greater influence.
• Connection to Evaluation: Metrics like Fleiss' Kappa can be calculated per item; items with low agreement (low per-item Kappa) may trigger a fallback to a weighted consensus from a trusted subset of models.
• Application: Central to mixture of experts architectures and federated learning aggregation protocols like Federated Averaging (FedAvg).