Faithfulness Metric

The core mechanism relies on Natural Language Inference (NLI) to classify the relationship between a source premise and a generated claim.

• Entailment: The claim is logically supported by the source.
• Contradiction: The claim directly opposes the source.
• Neutral: The claim introduces new, unsupported information.

This granular classification allows for fine-grained scoring beyond binary faithful/unfaithful labels, enabling precise identification of hallucination types.

Modern faithfulness metrics do not evaluate entire summaries at once. They first decompose the generated text into atomic facts—short, self-contained assertions that each express a single piece of information.

Each atomic fact is individually verified against the source document. This prevents a single hallucination from contaminating the score of an otherwise accurate summary and provides a more precise, granular measure of factual density.

A faithful statement must be fully grounded in the provided source text, not just factually true in the world. A model might state a true fact that is absent from the source, which constitutes a faithfulness violation.

• Intrinsic Faithfulness: Consistency with the provided input context only.
• Extrinsic Faithfulness: Consistency with external world knowledge (a separate, broader task).

Metrics for answer synthesis strictly measure intrinsic faithfulness to prevent the model from introducing unsourced knowledge.

A critical sub-task is identifying when a generated statement logically conflicts with the source. This goes beyond simple keyword matching to require logical inference.

For example, if the source states 'revenue grew by 10%' and the summary says 'revenue declined,' the metric must flag a direct contradiction. Advanced metrics use NLI models fine-tuned on contradiction datasets to catch these semantic inversions, which are among the most damaging types of hallucinations.

The ultimate validation for any automated metric is its correlation with human evaluators. Leading faithfulness metrics are benchmarked against expert annotations to ensure their scores reflect a genuine assessment of factual consistency.

• Pearson/Spearman correlation measures linear and rank agreement.
• Inter-annotator agreement establishes the human baseline ceiling.

A metric that achieves high correlation with human judgment can serve as a reliable, scalable proxy for manual review in production evaluation pipelines.

Faithfulness can be measured at multiple levels of granularity, each serving a different diagnostic purpose.

• Document-Level: A single holistic score for the entire summary.
• Sentence-Level: Scores for each sentence, identifying specific problematic spans.
• Fact-Level: The most granular, scoring each decomposed atomic fact.

Fact-level evaluation is the current state-of-the-art, providing actionable feedback for debugging and model improvement by pinpointing exactly which assertions are unsupported.