Inter-Annotator Agreement (IAA)

Cohen's Kappa measures the agreement between two annotators on a categorical scale, correcting for the agreement expected by chance. It is the standard metric for binary or multi-class classification tasks with exactly two annotators.

• Formula: κ = (p₀ - pₑ) / (1 - pₑ), where p₀ is the observed agreement and pₑ is the expected agreement by chance.
• Interpretation: Values range from -1 to 1. κ > 0.8 indicates almost perfect agreement; κ between 0.6 and 0.8 indicates substantial agreement.
• Primary Use Case: Validating label quality for sentiment analysis, topic classification, or named entity recognition tasks where two experts annotate the same dataset.

Fleiss' Kappa is a generalization of Cohen's Kappa for more than two annotators. It assesses the reliability of agreement across multiple raters for categorical items, making it essential for crowdsourced labeling projects.

• How it works: It calculates the degree of agreement over and above what would be expected by chance, based on the proportion of assignments to each category.
• Key Advantage: It can handle a variable number of annotators per item, which is common in platforms like Amazon Mechanical Turk.
• Primary Use Case: Measuring consensus in large-scale data labeling initiatives, such as image classification or content moderation, where multiple crowd workers label each sample.

Krippendorff's Alpha is a highly versatile reliability coefficient that works with any number of annotators, any scale of measurement (nominal, ordinal, interval, ratio), and can handle missing data. It is considered one of the most robust IAA metrics.

• Flexibility: It can measure agreement for text spans (interval), ranked preferences (ordinal), or simple categories (nominal).
• Missing Data: It gracefully handles datasets where not every annotator labeled every item.
• Primary Use Case: Complex annotation tasks like semantic textual similarity scoring (interval), coreference resolution (nominal), or sentiment intensity ranking (ordinal). It is the metric of choice for establishing reliability in academic NLP research.

The Intraclass Correlation Coefficient measures agreement for continuous or ordinal data, assessing both the correlation and the absolute agreement between annotators. It is crucial for tasks where the magnitude of a rating matters.

• Variants: ICC(1,1) for single rater reliability; ICC(3,k) for fixed set of k raters' average reliability.
• Interpretation: Values closer to 1 indicate high reliability. ICC > 0.75 is often considered excellent.
• Primary Use Case: Annotating subjective but continuous scores, such as toxicity severity (0-100), translation quality assessment, or audio sentiment intensity. It is widely used in psychology, medicine, and subjective evaluation benchmarks.

Percent Agreement is the simplest IAA metric, calculated as the number of items where annotators agree divided by the total number of items. While easy to compute, it is misleading as it does not account for chance agreement.

• Limitation: In tasks with high class imbalance or few categories, chance agreement (pₑ) can be very high, inflating the perceived reliability.
• Appropriate Use: It can serve as a quick, preliminary sanity check, but should not be the sole metric for reporting data quality in formal evaluations.
• Example: For a binary task with 90% positive examples, two annotators randomly guessing 'positive' would have a 82% agreement by chance alone, making a raw 85% agreement score poor, not good.

The upper bound of model performance is fundamentally constrained by IAA. A model's confidence scores are calibrated against the 'ground truth,' but if the human-provided labels have low agreement, this truth is fuzzy, making perfect model accuracy impossible and confidence calibration flawed.

• Key Principle: Model accuracy on a test set cannot statistically exceed the level of human agreement on that set.
• Use in Evaluation: IAA establishes the inherent difficulty and label noise ceiling of a dataset. A low IAA score signals that the task is ambiguous or instructions are poor, necessitating task redesign before model training.
• Practical Implication: Before deploying a confidence scoring system, measure IAA. If annotators disagree, the model will be overconfident on inherently ambiguous examples. High IAA provides a solid foundation for meaningful uncertainty quantification and selective classification.

A confidence score is a probabilistic measure, typically derived from a model's final output layer (e.g., softmax), that quantifies the model's self-assessed certainty in the correctness of a specific prediction. It is the primary internal metric for an AI's self-evaluation.

• Purpose: Provides a scalar value (e.g., 0.95) indicating trust in a single output.
• Contrast with IAA: While IAA measures agreement between humans, a confidence score reflects a single model's internal belief. A well-calibrated model's confidence should correlate with human agreement rates.

Uncertainty Quantification (UQ) is the machine learning subfield focused on measuring and interpreting the different types of uncertainty inherent in a model's predictions. It provides a more nuanced view than a single confidence score.

• Aleatoric Uncertainty: Irreducible noise inherent in the data (e.g., ambiguous image, conflicting expert labels). High aleatoric uncertainty suggests low IAA is expected.
• Epistemic Uncertainty: Reducible uncertainty from a lack of model knowledge (e.g., unfamiliar data). High epistemic uncertainty suggests the model needs more training data.
• Relation to IAA: IAA directly measures the aleatoric uncertainty present in the labeling task itself.

Calibration error measures the discrepancy between a model's predicted confidence scores and its actual empirical accuracy. A perfectly calibrated model that predicts a confidence of 0.8 should be correct 80% of the time.

• Expected Calibration Error (ECE): A common metric that bins predictions by confidence and computes the average gap between bin accuracy and bin confidence.
• Benchmarking: IAA establishes the upper limit of achievable accuracy for a task, against which model calibration is measured. If human annotators only agree 85% of the time, a model cannot be perfectly calibrated above 85% confidence.

Selective classification, or classification with a rejection option, is a paradigm where a model is allowed to abstain from making a prediction on inputs where its confidence is below a chosen threshold.

• Use Case: Critical for high-stakes applications where incorrect predictions are costly.
• Trade-off: Illustrated by a risk-coverage curve, which plots error rate against the fraction of samples the model chooses to predict on.
• IAA's Role: The optimal confidence threshold for abstention is often set relative to the task's inherent difficulty, as quantified by IAA metrics like Cohen's Kappa.

Out-of-Distribution (OOD) Detection is the task of identifying whether a given input sample is statistically different from the data distribution the model was trained on.

• Critical Safety Mechanism: Models often make overconfident, incorrect predictions on OOD data.
• Connection to IAA: OOD samples represent a form of epistemic uncertainty. A robust system uses OOD detection to flag inputs where the model's confidence score is likely untrustworthy, similar to how low IAA flags data where human labels are unreliable.

Self-consistency is a decoding strategy for complex reasoning tasks (like chain-of-thought) where multiple reasoning paths are sampled, and the final answer is determined by a majority vote over the generated outputs.

• Agreement as Confidence: The degree of agreement among sampled outputs is used as a proxy for answer confidence.
• Analogy to IAA: This is a form of intra-model agreement, analogous to IAA but performed by a single model with stochasticity. High self-consistency correlates with higher answer correctness, mirroring how high IAA correlates with higher label reliability.