RAGTruth

RAGTruth employs a two-level annotation hierarchy that captures hallucinations with exceptional precision:

• Passage-Level Hallucination: Labels entire generated spans as hallucinated when they contain unsupported claims
• Word-Level Hallucination: Pinpoints the exact tokens within a passage that constitute the factual error

This dual granularity enables both coarse filtering of hallucinated responses and fine-grained error analysis for model debugging. Annotators are trained to distinguish between intrinsic hallucinations (contradicting the source) and extrinsic hallucinations (adding unsupported information).

The corpus spans four distinct domains to test generalization across knowledge types:

• Wikipedia: Encyclopedic factual knowledge with high entity density
• News: Time-sensitive information requiring temporal reasoning
• Medical: Domain-specific terminology and causal relationships from PubMed abstracts
• Finance: Numerical precision and quantitative claims from earnings reports

Each domain presents unique hallucination patterns. Medical texts often exhibit relation-level errors where causal links are fabricated, while finance texts show numerical hallucinations where figures are altered or invented.

Unlike general hallucination benchmarks, RAGTruth is designed specifically for retrieval-augmented pipelines. The evaluation protocol:

• Provides the retrieved context passages alongside the generated response
• Requires evaluators to verify each claim against the exact source text provided, not external knowledge
• Measures grounding fidelity—whether the model faithfully uses the retrieved information without fabrication

This isolates RAG-specific failures from general knowledge gaps, distinguishing between retrieval failures (wrong context) and generation failures (ignoring correct context).

All annotations in RAGTruth are produced by trained human annotators following rigorous guidelines:

• Inter-annotator agreement measured via Cohen's Kappa to ensure labeling consistency
• Multi-round adjudication where disagreements are resolved by senior annotators
• Span-level verification requiring annotators to highlight the exact source evidence for each claim

This human-validated ground truth provides a gold-standard evaluation set against which automated hallucination detection methods like SelfCheckGPT and NLI-based evaluators can be benchmarked.

RAGTruth includes a standardized evaluation harness for reproducible benchmarking:

• Pre-formatted test splits with consistent prompt templates across models
• Automated metric computation for hallucination rate, precision, and recall
• Baseline results published for major LLMs including GPT-4, Claude, and Llama variants

The benchmark supports both black-box API evaluation and white-box model inspection, enabling researchers to correlate hallucination rates with model architecture choices and retrieval configurations.

RAGTruth annotations map to a structured hallucination taxonomy that classifies errors by type:

• Entity-Level: Invented people, places, or organizations not in the source
• Relation-Level: Fabricated connections between entities that appear in the source
• Numerical: Altered statistics, dates, or quantitative values
• Temporal: Incorrect chronological ordering or time references

This taxonomy enables granular risk analysis, allowing practitioners to identify which error types dominate in their RAG pipeline and target mitigation strategies accordingly.