Faithfulness Metric

The most rigorous faithfulness metrics decompose generated text into atomic claims and use Natural Language Inference (NLI) models to verify each claim against the source document. A claim is considered faithful only if the NLI model classifies the relationship as entailment—meaning the source text logically implies the claim. This approach provides fine-grained, interpretable scores by identifying exactly which statements are unsupported or contradicted, moving beyond surface-level n-gram overlap to assess logical consistency.

This paradigm, exemplified by metrics like QuestEval, reframes faithfulness evaluation as a question-answering task. The system generates questions from the summary, then attempts to answer them using only the source document. The answer overlap between summary-derived answers and source-derived answers produces a precision and recall score. This method excels at detecting subtle hallucinations where wording differs but meaning should be preserved, as it tests whether the same factual information can be extracted from both texts.

Traditional surface-level metrics like ROUGE and BLEU measure lexical overlap between generated text and a reference, but they are not valid faithfulness metrics. A summary can achieve high ROUGE scores by copying source phrases verbatim while simultaneously containing hallucinated entities or relationships. These metrics fail to distinguish between faithful paraphrasing and unfaithful fabrication because they lack semantic understanding. Modern evaluation frameworks explicitly deprecate n-gram overlap for factual grounding assessment in favor of NLI-based or QA-based methods.

Advanced faithfulness evaluation introduces counterfactual perturbations to source documents—such as negating a key fact or swapping an entity—and measures whether the metric correctly identifies the resulting generated text as unfaithful. A robust metric must be sensitive to these semantic manipulations while remaining invariant to paraphrasing. This testing methodology validates that a metric is not merely a stylistic similarity checker but genuinely evaluates logical entailment and factual grounding.

Faithfulness metrics specifically target extrinsic hallucinations—information in the generated text that cannot be found in or inferred from the provided source context. This is distinct from intrinsic hallucinations, which are internal contradictions within the generated text itself. Extrinsic detection requires the metric to have a strict closed-world assumption: the source document is the complete universe of admissible facts. Any entity, date, or relationship not directly supported by the source is flagged as unfaithful, regardless of its real-world truth.

The gold standard for validating any faithfulness metric is its correlation with human annotator judgments. Benchmarks like SummEval and FRANK provide human-annotated faithfulness scores across diverse summarization outputs. Top-performing metrics achieve Pearson correlations above 0.70 with human ratings by combining entailment models with fine-grained claim extraction. This alignment ensures that automated evaluation pipelines can reliably replace costly human review in production LLMOps workflows.