Human Evaluation (HITL)

A core methodology where human judges are presented with two outputs (e.g., from different models or configurations) and asked to select the preferred one based on specific criteria like fluency, helpfulness, or factual accuracy. This directly yields a win rate and establishes a robust preference ranking.

• Key Use: Ranking model versions, tuning hyperparameters, and evaluating subjective qualities.
• Example: In a chatbot evaluation, judges see two responses to the same user query and select the more coherent and helpful answer.

A psychometric method where evaluators rate a single AI output on an ordinal scale (e.g., 1-5 or 1-7) across multiple predefined dimensions. This provides granular, quantitative scores for subjective attributes.

• Common Dimensions: Factual Correctness, Coherence, Completeness, Toxicity, Instruction Following.
• Analysis: Requires calculating Inter-Annotator Agreement (e.g., Fleiss' Kappa) to ensure rating consistency and reliability before aggregating scores.

A diagnostic methodology where human experts systematically identify, classify, and tag specific failures in model outputs. This goes beyond a simple score to provide actionable insights for model improvement.

• Common Error Types: Hallucinations (fabricated facts), Contradictions, Off-Topic Responses, Repetition, Safety Violations.
• Output: A labeled dataset of failures used to train hallucination detection classifiers or to guide red teaming efforts.

A proactive, security-inspired methodology where human testers (the "red team") deliberately craft challenging or malicious inputs to probe for model vulnerabilities, biases, or safety failures.

• Objective: To expose weaknesses before deployment, assessing robustness and alignment with ethical guidelines.
• Focus Areas: Jailbreaking (circumventing safety filters), prompt injection, generating harmful content, and exploiting reasoning flaws.

An end-to-end evaluation where human judges assess whether an AI agent or system successfully completes a complex, multi-step task in a simulated or real environment. This measures practical utility.

• Examples: Evaluating an agentic system's ability to book travel correctly based on email constraints, or a coding assistant's success in fixing a bug in a codebase.
• Metrics: Binary success/failure, time to completion, and number of required human interventions (HITL loops).

Not an evaluation method itself, but the critical statistical process for validating the reliability of any human evaluation. It quantifies the consistency of judgments across multiple annotators.

• Primary Metrics: Fleiss' Kappa (multiple annotators, categorical labels), Cohen's Kappa (two annotators), and Intraclass Correlation Coefficient (ICC) for continuous ratings.
• Purpose: Low agreement indicates poorly defined guidelines, ambiguous tasks, or unreliable data, invalidating the evaluation results.

The verified, accurate data or labels used as the definitive reference for training and, critically, for evaluating the performance of a machine learning model. Human evaluation often creates or validates ground truth.

• Role in HITL: Human judges provide the authoritative labels (e.g., 'correct/incorrect', quality score) that become the benchmark for automated metrics.
• Challenge: Establishing high-quality, unbiased ground truth is expensive and foundational; errors here propagate through all downstream evaluation.

A comparative evaluation metric derived from pairwise comparisons. It measures the percentage of times one model's output is preferred over another's by human or automated judges.

• Methodology: Judges are shown two blind outputs (A/B) for the same prompt and select the winner. A model's win rate is calculated across many such comparisons.
• Advantage: Directly measures human preference, which can correlate better with perceived quality than automated scores like BLEU or ROUGE.
• Use Case: Commonly used to rank conversational AI assistants or text-generation models.

The fundamental evaluation methodology where a judge is presented with two outputs (e.g., from different model versions or systems) for the same input and is asked to select the preferred one.

• Output: Generates a preference label (A > B, B > A, Tie).
• Scalability: Can be crowdsourced (e.g., via platforms like Scale AI or Amazon Mechanical Turk) to gather large volumes of human judgments.
• Statistical Analysis: Results are aggregated using methods like the Bradley-Terry model to produce a global ranking of models from many pairwise contests.