Inter-Annotator Agreement (IAA) for Traces

Cohen's Kappa is a statistic that measures the agreement between two raters for categorical items, correcting for the agreement expected by random chance. It is the standard metric for binary or multi-class classification of trace steps (e.g., labeling a step as 'Correct', 'Incorrect', or 'Irrelevant').

• Interpretation: Values range from -1 to 1. κ > 0.8 indicates almost perfect agreement; κ between 0.6 and 0.8 is substantial; κ < 0.4 indicates poor agreement.
• Use Case: Ideal for evaluating if annotators agree on the categorical label of individual reasoning steps, such as detecting hallucinations or classifying logical fallacies.

Fleiss' Kappa is a generalization of Cohen's Kappa for three or more raters. It assesses the reliability of agreement across a fixed number of annotators, each classifying items into mutually exclusive categories.

• Key Feature: It calculates the degree of agreement over and above what would be expected by chance for multiple raters, providing a single score for the entire annotation pool.
• Use Case: Essential for large-scale trace evaluation projects where more than two experts must label thousands of reasoning steps, ensuring the annotation schema itself is not ambiguous.

The Intraclass Correlation Coefficient measures agreement for continuous or ordinal ratings (e.g., scoring a trace's coherence from 1-7). It assesses both the correlation and the absolute agreement between raters.

• Variants: ICC(1,1) for single-rater reliability; ICC(3,k) for fixed raters and mean score reliability.
• Interpretation: Values close to 1 indicate high reliability. It is more appropriate than Pearson correlation for agreement, as it penalizes systematic rater bias (e.g., one rater consistently scoring 2 points higher than another).
• Use Case: Validating scores for metrics like Stepwise Coherence or Trace Validity on a Likert scale.

Krippendorff's Alpha is a robust agreement coefficient designed to handle multiple raters, missing data, and any level of measurement (nominal, ordinal, interval, ratio). It is based on the observed versus expected disagreement.

• Key Strength: Its flexibility makes it suitable for complex, real-world annotation tasks where not every rater evaluates every trace, or where labels are ordinal scores.
• Benchmark: α ≥ 0.800 is required to draw substantive conclusions from the data; α ≥ 0.667 permits tentative conclusions.
• Use Case: The gold standard for complex trace evaluation studies with large, partially overlapping teams of annotators using rich annotation schemas.

Percent Agreement is the simplest IAA metric, calculated as the number of times raters agree divided by the total number of items rated. It does not correct for chance agreement.

• Limitation: Can be misleadingly high for tasks with skewed category distributions (e.g., if 95% of steps are 'Correct', random guessing yields 90%+ agreement).
• Appropriate Use: Serves as a quick, preliminary sanity check or for reporting alongside chance-corrected metrics like Kappa. It is most informative when category prevalence is balanced.

Weighted agreement metrics (e.g., Weighted Kappa) account for the degree of disagreement between ordinal ratings. Not all disagreements are equal; mistaking a '5' for a '4' is less severe than mistaking a '5' for a '1'.

• Mechanism: A weight matrix (linear, quadratic) assigns partial credit for near-misses and full penalty for major disagreements.
• Use Case: Critical for scoring ordinal rubrics in trace evaluation, such as a 5-point scale for logical soundness. It provides a more nuanced view of rater consistency than unweighted metrics.

A statistical measure of inter-annotator agreement that accounts for the agreement expected by chance. It is the standard metric for categorical labels on traces (e.g., 'correct', 'hallucination', 'incomplete').

• Calculation: κ = (Pₐ - Pₑ) / (1 - Pₑ), where Pₐ is the observed agreement and Pₑ is the expected agreement.
• Interpretation: Values range from -1 to 1. κ > 0.8 indicates excellent agreement, κ between 0.6-0.8 is substantial, and κ < 0.4 indicates poor reliability.
• Use Case: Essential for validating a trace annotation schema where evaluators classify each reasoning step into predefined categories.

A generalization of Cohen's Kappa used to measure agreement among three or more annotators when assigning categorical ratings to reasoning traces. It is robust for large-scale evaluation studies.

• Key Difference: Unlike Cohen's Kappa which is pairwise, Fleiss' Kappa calculates a single statistic for the entire group of raters.
• Application: Ideal for benchmarking studies where multiple experts independently score the logical consistency or trace validity of the same set of agent outputs.
• Requirement: All annotators must evaluate the exact same set of traces.

A reliability metric used for continuous or ordinal scores (e.g., a stepwise coherence score from 1-5). It assesses both the consistency and absolute agreement between raters.

• Common Models: ICC(2,1) for absolute agreement of a single rater's score; ICC(3,k) for consistency when using the mean of k raters.
• Advantage: More appropriate than Kappa for Likert-scale ratings of trace quality, such as scoring the clarity of a tool-use rationale.
• Output: Ranges from 0 to 1, with values above 0.75 indicating good reliability.

A versatile reliability coefficient that works with any number of annotators, any metric scale (nominal, ordinal, interval, ratio), and can handle missing data. It is a highly robust measure for complex annotation tasks.

• Flexibility: Can be applied to diverse trace annotation tasks, from coding cognitive biases (nominal) to rating specification compliance (ordinal).
• Benchmark: α ≥ 0.800 is required to draw substantive conclusions from the data; α ≥ 0.667 permits tentative conclusions.
• Utility: The preferred metric for establishing reliability in content analysis research, directly applicable to red-teaming trace evaluation.

The simplest measure of IAA, calculated as the percentage of items (e.g., reasoning steps) for which all annotators assign the identical label or score.

• Limitation: Does not correct for chance agreement, thus often inflates reliability estimates. A high percent agreement with a low Kappa indicates labels are easy to guess.
• Appropriate Use: Can provide a quick, intuitive baseline, but should always be reported alongside a chance-corrected metric like Cohen's Kappa for scientific rigor.
• Example: If 90 out of 100 reasoning steps receive the same hallucination detection label from two raters, percent agreement is 90%.

The process of creating detailed, unambiguous instructions and scoring criteria for human evaluators. This is a prerequisite for achieving high IAA.

• Components: Includes clear definitions, positive/negative examples, decision trees for edge cases, and practice calibration sets.
• Iterative Process: Guidelines are refined over multiple rounds of pilot annotation and IAA calculation until acceptable reliability (e.g., κ > 0.7) is achieved.
• Critical For: Defining trace validity, stepwise coherence, and other subjective qualities in an operationalizable way. Poor guidelines guarantee low IAA, rendering the evaluation unreliable.